Source-linked AI summary
A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, Roman Wang, Nikhil Singh, Taylor L. Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, Gilbert Strang
TL;DR
The paper addresses limited performance of text-only neural networks on university-level mathematics while seeking systems that can solve, explain, and generate course problems. It uses Codex, pre-trained on text and fine-tuned on code, to synthesize and execute programs with few-shot learning. The system reaches human-level performance across solving, explanation, and generation tasks, including 81% automatic accuracy on university mathematics questions.
Problem
Text-only neural networks had failed on advanced mathematics problems, leaving university-level solving, explanation, and question generation insufficiently demonstrated at scale.
Method
The paper uses Codex to synthesize and execute programs from mathematics questions, then generates explanations and new questions through an automated pipeline.
Results
The system solves university mathematics problems at 81% automatic accuracy and produces explanations and new questions rated similarly to human-written material.
Takeaways & Limitations
Program synthesis provides a scalable representation and computation environment for solving, explaining, and generating university-level mathematics problems.
Takeaways & Limitations
Some questions require manual editing because they are vague or contain redundant contextual information, while proofs, non-text modalities, and computationally intractable problems are outside the supported setting.
Abstract
from arXiv · showhide
We demonstrate that a neural network pre-trained on text and fine-tuned on code solves mathematics course problems, explains solutions, and generates new questions at a human level. We automatically synthesize programs using few-shot learning and OpenAI's Codex transformer and execute them to solve course problems at 81% automatic accuracy. We curate a new dataset of questions from MIT's largest mathematics courses (Single Variable and Multivariable Calculus, Differential Equations, Introduction to Probability and Statistics, Linear Algebra, and Mathematics for Computer Science) and Columbia University's Computational Linear Algebra. We solve questions from a MATH dataset (on Prealgebra, Algebra, Counting and Probability, Intermediate Algebra, Number Theory, and Precalculus), the latest benchmark of advanced mathematics problems designed to assess mathematical reasoning. We randomly sample questions and generate solutions with multiple modalities, including numbers, equations, and plots. The latest GPT-3 language model pre-trained on text automatically solves only 18.8% of these university questions using zero-shot learning and 30.8% using few-shot learning and the most recent chain of thought prompting. In contrast, program synthesis with few-shot learning using Codex fine-tuned on code generates programs that automatically solve 81% of these questions. Our approach improves the previous state-of-the-art automatic solution accuracy on the benchmark topics from 8.8% to 81.1%. We perform a survey to evaluate the quality and difficulty of generated questions. This work is the first to automatically solve university-level mathematics course questions at a human level and the first work to explain and generate university-level mathematics course questions at scale, a milestone for higher education.
Introduction
The paper presents Codex, a transformer pre-trained on text and fine-tuned on code, as a system that synthesizes programs to solve university mathematics problems and supports explanations and question generation. Across sampled university courses and benchmark problems, the approach substantially improves automatic solution accuracy over prior methods.
- Introduction: Unlike earlier approaches focused on elementary or course-specific numeric answers, program synthesis targets a broader range of university-level mathematics problems.The paper contrasts executable programs with expression-tree methods that do not extend to university-level courses or generalize across courses.
- Introduction: 81% of randomly selected university-level mathematics problems were automatically solved by Codex using program synthesis and few-shot learning.The sample covered six MIT mathematics courses and one Columbia University course.
- Introduction: The system also automatically explains synthesized solutions and generates new mathematics questions, with outputs including equations, Boolean values, plots, numerical values, and vectors.Codex receives the question and generated code to produce step-by-step explanations.
- Introduction: 81.1% accuracy on the MATH benchmark improved on the previous state-of-the-art result of 8.8% using automatic few-shot learning.The benchmark contains challenging problems sourced from high-school mathematics competitions.
- Introduction: The approach combines text pre-training with code fine-tuning, zero-shot or few-shot prompting, and retrieval of solved question–code pairs using cosine similarity.These components support synthesis of programmatic solutions without additional model training.
Methods
The study samples university mathematics and MATH benchmark questions, then uses Codex to synthesize executable programs, solve questions, produce multiple answer modalities, and generate explanations and new questions.
- Dataset: The dataset randomly samples 25 questions from each of seven university courses and 15 questions from six MATH topics.The university courses include six MIT courses and Columbia University's Computational Linear Algebra; sampled MATH topics include Algebra, Counting & Probability, Intermediate Algebra, Number Theory, and Prealgebra.
- Program synthesis: 81% of course questions and 81.1% of MATH questions are automatically solved using few-shot learning with synthesized programs.The remaining 19% of course questions and 18.9% of MATH questions are manually prompted.
- Evaluation: The workflow compares automatic solving with GPT-3 and Codex and reports standard errors for total solve-rate estimates.Figure 4 compares zero-shot and few-shot approaches across courses and the MATH benchmark.
- Outputs: Example outputs include equations, Boolean values, plots, numerical values, and vectors across the mathematics courses.The pipeline executes synthesized programs to produce answers in multiple modalities.
- Prompt context: Prompts specify Python and, where useful, libraries such as SymPy; course-specific package usage is incorporated into program synthesis.All courses use NumPy and SymPy, while Matplotlib supports plotting questions.
- Few-shot learning: Few-shot learning retrieves similar solved questions through question embeddings and supplies question–code pairs as examples.Zero-shot synthesis solves 71% of questions before the few-shot stage increases the automatic solve rate to 81%.
- Explanations: Explanations use the original question, generated code, and an automatic prompt to produce a step-by-step explanation of the solution code.The question is retained because generated code may be a lossy representation of the original question.
- Question generation: Codex generates new course questions by extending randomly truncated numbered lists of human-written questions.The process is repeated to create many new questions for each course.
Question Generation and their Human Evaluation.
The paper generates course questions and evaluates them through student surveys, while figures illustrate program-based solution and explanation workflows across mathematics courses.
- Human evaluation: Students who had taken the courses compared machine-generated and human-written questions for quality and difficulty.The survey was approved by the MIT IRB and included informed consent.
- Workflow: Figure 3 shows pipelines that transform questions into synthesized programs, executed answers, and generated explanations.Examples span single-variable and multivariable calculus, differential equations, probability and statistics, and linear algebra.
- Human evaluation: The survey randomly mixed five original and five generated questions for each of six MIT courses.Students reviewed ten questions per course without being told each question's source.
- Automatic solving: Figure 4 compares GPT-3 and Codex solve rates across MIT and Columbia courses and the MATH benchmark.Codex reaches 71.1% zero-shot and 81.1% few-shot on courses, compared with GPT-3's 18.8% zero-shot and 30.8% few-shot with CoT.
- Survey design: For each of 60 questions, students classified authorship, judged course appropriateness, and rated difficulty from 1 to 5.Students provided ratings rather than solving the questions, and the anonymous survey was conducted online.
Results
The study evaluates Codex on university mathematics problems, generated questions, and question embeddings. Codex achieves high automatic solving accuracy, while survey results show generated questions are similar in difficulty but somewhat less course-appropriate.
- Questions Solved: 213 of 265 sampled questions were solved automatically across seven university courses and six MATH topics.The sample included 25 questions per university course and 15 questions per MATH topic.
- Automatically Generating New Questions: New questions were generated for each course and MATH topic using prompts containing numbered human-written questions.The study presents 130 generated questions, including ten for each of seven courses and six MATH topics.
- Student Survey Results: Survey participants rated machine-generated and human-written questions similarly difficult within confidence intervals.Difficulty was rated on a scale from 1 to 5, with Figure 7 reporting means and 95% confidence intervals.
- Student Survey Results: Human-written questions were rated slightly more appropriate for their courses than machine-generated questions.The survey compared course appropriateness for human-written and machine-generated questions.
- Student Survey Results: Machine-generated questions were rated equally likely to be machine-generated and human-written by survey participants.This comparison appears in the right side of Figure 7 Panel C.
- Questions Solved: 81% automatic accuracy was achieved on university mathematics course problems using Codex and program synthesis with few-shot learning.Programs provide step-by-step commands, making chain-of-thought reasoning inherent in the generated programs.
- Human Level: Embedding visualization produced distinctive clusters corresponding to course topics, including linear algebra and probability-and-statistics groups.The 175 questions were embedded in 2,048 dimensions and reduced to two dimensions with UMAP.
Implementation Details.
The implementation uses few-shot prompts that pair questions with code and chain-of-thought prompts for GPT-3. The paper also identifies problem types that the model cannot solve.
- Prompting: Few-shot prompts place each question in a docstring followed by code, then append the target question in the same format.Codex is prompted after the question–code examples and target question are inserted.
- Prompting: GPT-3 chain-of-thought prompts append “Let’s think step by step.” after the few-shot questions and answers.The supplied passages describe this as the chain-of-thought prompt construction.
- Reproducibility: The paper states that data and code are publicly available at the mathQ repository.The repository URL is given in the implementation materials.
- Scope Boundaries: The model is incapable of solving questions involving images or other non-text modalities, proof-required solutions, or computationally intractable problems.The paper gives factoring very large primes as an example of computational intractability.
Conclusion
The paper argues that code-based few-shot program synthesis enables a single neural network to solve, explain, and generate university-level mathematics problems at human level. It attributes this breadth partly to the flexibility and readability of programs and reports positive student evaluations of generated questions.
- Conclusion: Few-shot program synthesis using OpenAI Codex solves, explains, and generates university-level mathematics problems at a human level.The approach uses a neural network pretrained on text and fine-tuned on code.
- Conclusion: Programs offer broader flexibility than expression trees through abstraction, modularity, and high-level logic.The paper also describes program outputs as more human-readable and capable of expressing logical deductions through comments, function names, and variable names.
- Conclusion: Matplotlib enables the system to produce equation graphs, an ability the paper describes as time-consuming for humans and pedagogically beneficial.
- Conclusion: Generated questions were rated by students as equally likely to be human-written, similarly difficult, and appropriate for their courses.The paper also reports scaling the work to over thirty STEM courses across thirteen departments.
A. Solutions for MIT 18.01: Single Variable Calculus
The MIT 18.01 examples show Codex generating executable programs that solve calculus questions and produce explanations, numerical outputs, and plots. Several examples contrast these outputs with incorrect GPT-3 answers.
- A. Solutions for MIT 18.01: Single Variable Calculus: The course syllabus covers single-variable differentiation, integration, differential equations, applications, polar coordinates, and integration techniques.
- A. Solutions for MIT 18.01: Single Variable Calculus: Codex automatically solves limit questions by generating code that uses SymPy’s limit function with the target expression, variable, and limiting value.
- A. Solutions for MIT 18.01: Single Variable Calculus: Codex explanations describe computational steps such as defining functions, differentiating, finding roots, and plotting results.One example uses Newton’s method to find roots after differentiating a polynomial.
- A. Solutions for MIT 18.01: Single Variable Calculus: Codex programs implement trapezoidal, Simpson’s, and Gauss-Legendre numerical integration methods and print their results.The workflow can also plot the function used in the integration problem.
- A. Solutions for MIT 18.01: Single Variable Calculus: GPT-3 incorrectly estimated the area under a curve from 1 to 2 as approximately 1.4 in one calculus example.
C. Solutions for MIT 18.03: Differential Equations
The MIT 18.03 examples use Codex-generated programs to analyze differential-equation systems, calculate periods, and create plots. The examples include both correct and incorrect GPT-3 evaluations.
- C. Solutions for MIT 18.03: Differential Equations: The syllabus includes analytical, graphical, and numerical methods for differential equations, alongside linear systems, eigenvalues, oscillations, and Fourier series.
- C. Solutions for MIT 18.03: Differential Equations: Codex workflows for differential equations generate grids, vector fields, critical points, Jacobians, eigenvalues, and streamplots.These steps support analysis and visualization of dynamical systems.
- C. Solutions for MIT 18.03: Differential Equations: GPT-3 correctly identified the critical point as (0,0) in one differential-equation example.
- C. Solutions for MIT 18.03: Differential Equations: GPT-3 incorrectly classified the critical point at (0,0) as a saddle point in another system example.The Codex explanation computed the Jacobian and used eigenvalue-based analysis before producing the classification.
- C. Solutions for MIT 18.03: Differential Equations: Codex calculated the period of sin(3t) as 2π/3, reported numerically as 2.0943951023931953.The workflow creates a time array, plots the function, calculates the period, and prints it.
D. Solutions for MIT 18.05: Introduction to Probability and Statistics
The MIT 18.05 examples pair Codex-generated programs with explanations and compare their outputs against GPT-3 responses. The examples span probability simulations, statistical calculations, and distributions, with GPT-3 responses frequently evaluated as incorrect.
- The section covers probability models, combinatorics, random variables, distributions, estimation, testing, confidence intervals, and regression.
- Probability simulations: Automatic Codex programs simulate dice and card experiments, including win probabilities and three-pair probabilities.The simulations define random trials, count successful outcomes, and return empirical frequencies.
- Probability simulations: GPT-3 incorrectly answers both dice-comparison examples with probability 3/5.
- Statistical calculations: For a sampling-probability calculation, GPT-3 applies a normal approximation and z-score but is evaluated as incorrect.The response uses mean 30%, sample size 400, standard deviation 0.25, z = -0.8, and probability 0.22.
E. Solutions for MIT 18.06: Introduction to Linear Algebra
The section presents MIT 18.06 linear algebra questions solved through Codex-generated programs, with explanations and outputs spanning eigenvalues, singular values, projections, and related computations.
- MIT 18.06 covers matrix theory and linear algebra, including systems, vector spaces, determinants, eigenvalues, singular value decomposition, and applications.
- Codex workflows pair the original question with synthesized code, executed output, and an explanation of the solution program.The workflow can produce equations, Boolean values, and plots as answers.
- The projection example correctly computes the projection of b onto a as [0;0] and plots the input vectors and projection.
- The linear algebra examples also evaluate eigenvalues, eigenvectors, singular values, and singular vectors through matrix computations.
- GPT-3’s response for one eigenvalue-and-singular-value example is evaluated as incorrect.
F. Solutions for MIT 6.042: Mathematics for Computer Science
The section applies program-based solutions and explanations to mathematics-for-computer-science problems involving discrete mathematics, probability, graph coloring, and combinatorial enumeration.
- MIT 6.042 emphasizes discrete mathematics and proof techniques for computer science, including graphs, induction, recurrences, algorithms, number theory, counting, and probability.
- A five-card enumeration program computes the probability that a hand contains four of a kind as 0.00024.The program constructs all five-card hands, filters four-of-a-kind hands, and divides the counts.
- A graph-coloring example manually produces chromatic number 3, and GPT-3’s response is evaluated as correct.
- Codex explanations describe probability calculations, recursive counting procedures, and object methods in step-by-step form.
- One GPT-3 response in the section is evaluated as incorrect.
H. Solutions for MATH: Prealgebra
The supplied examples span MATH benchmark topics from foundational arithmetic and algebra through number theory, with programs and explanations solving selected problems.
- Prealgebra and related topics: MATH Prealgebra problems include means, medians, modes, primes, divisibility, fractions, decimals, ratios, equations, inequalities, and counting.
- Prealgebra and related topics: A prime-testing program checks small cases, divisibility by 2 and 3, and possible factors through the square root of n.
- Algebra and number theory: The supplied MATH examples also cover algebra topics such as exponents, logarithms, functions, graphs, and quadratic equations.
- Algebra and number theory: Number Theory examples address primes, factorization, greatest common divisors, least common multiples, divisors, bases, and modular arithmetic.
P. Generation of MIT 18.03: Differential Equations
The section includes generated-question tables for MIT differential-equations material alongside examples spanning differential equations, Laplace transforms, numerical integration, and numerical solution methods.
- Generated questions: The section includes a table of new questions generated from MIT 18.03 Differential Equation questions and their closest existing questions.
- Differential-equation tasks: The examples ask for general and particular solutions, direction fields, inverse Laplace transforms, and separable-variable solutions.
- Numerical methods: Numerical-method examples use Newton-forward differences, Runge-Kutta integration, the trapezoidal rule, and improved Euler approximation.
- Applications: The supplied material also contains an electrical-discharge problem involving battery energy over the first four hours.
- Generated questions: Additional tables list generated questions for MIT 18.05 probability and statistics and MIT 18.06 linear algebra.
S. Generation of MIT 6.042: Mathematics for Computer Science
Table 273 presents newly generated questions from MIT 6.042 Mathematics for Computer Science alongside their closest existing questions.
- The table lists new questions generated from the 6.042 dataset.
- Each generated question is paired with the closest question among the existing questions.
- The table is organized around comparison between generated and existing 6.042 questions.
T. Generation of Columbia University COMS3251: Computational Linear Algebra
The generation section presents newly generated questions for Columbia's Computational Linear Algebra dataset and several MATH topics, pairing them with closest existing questions. The listed examples span linear algebra tasks such as inverses, decompositions, projections, vector combinations, and subspace dimensions.
- T. Generation of Columbia University COMS3251: Computational Linear Algebra: Table 274 pairs newly generated COMS3251 Computational Linear Algebra questions with their closest existing questions.
- T. Generation of Columbia University COMS3251: Computational Linear Algebra: The listed linear algebra examples ask for left inverses, vector combinations, LU decompositions, matrix inverses, projections, and subspace dimensions.
- T. Generation of Columbia University COMS3251: Computational Linear Algebra: Additional examples ask for combinations yielding a vector or the zero vector and projections onto a matrix's column space.
- T. Generation of Columbia University COMS3251: Computational Linear Algebra: Tables 275–280 similarly pair generated MATH questions with closest existing questions across Algebra, Counting & Probability, Intermediate Algebra, Number Theory, Prealgebra, and Precalculus.