PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Assisted Generation of Difficult Math Questions

Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, Anirudh Goyal

arXiv:2407.21009v4cs.AIcs.LG

TL;DR

High-quality, varied, difficult math questions are becoming scarce, while human-only creation is not scalable and LLM-generated questions often fall short. The paper combines LLM generation from paired mathematical skills with human review; its resulting MATH2 questions are harder for leading models and more effective MATH in-context examples than examples from MATH.

  • Problem

    High-quality, varied, difficult math questions are becoming scarce, and benchmark-version performance differences raise concerns about evaluation saturation and overfitting.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    The framework uses LLMs to generate questions combining distinct mathematical skills and has humans screen and refine the resulting questions.

    Source: §1.2 · p. 3, §2 · p. 5

  • Results

    MATH2 is significantly more challenging for leading models, while its questions improve MATH performance more effectively as in-context examples than examples sourced from MATH.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The framework may also support efficient generation of high-quality data in domains beyond mathematics.

    Source: §5 · p. 9

    Initial LLM-generated questions often resembled existing questions, contained errors, or equated difficulty with tedious calculation.

    Source: §1.2 · p. 2

Abstract

from arXiv · show

Current LLM training positions mathematical reasoning as a core capability. With publicly available sources fully tapped, there is unmet demand for diverse and challenging math questions. Relying solely on human experts is both time-consuming and costly, while LLM-generated questions often lack the requisite diversity and difficulty. We present a design framework that combines the strengths of LLMs with a human-in-the-loop approach to generate a diverse array of challenging math questions. We leverage LLM metacognition skills [Didolkar et al., 2024] of a strong LLM to extract core "skills" from existing math datasets. These skills serve as the basis for generating novel and difficult questions by prompting the LLM with random pairs of core skills. The use of two different skills within each question makes finding such questions an "out of distribution" task for both LLMs and humans. Our pipeline employs LLMs to iteratively generate and refine questions and solutions through multiturn prompting. Human annotators then verify and further refine the questions, with their efficiency enhanced via further LLM interactions. Applying this pipeline on skills extracted from the MATH dataset [Hendrycks et al., 2021] resulted in MATH$^2$ - a dataset of higher-quality math questions, as evidenced by: (a) Lower performance of all models on MATH$^2$ than on MATH (b) Higher performance on MATH when using MATH$^2$ questions as in-context examples. Although focused on mathematics, our methodology seems applicable to other domains requiring structured reasoning, and potentially as a component of scalable oversight. Also of interest is a striking relationship observed between models' performance on the new dataset: the success rate on MATH$^2$ is the square on MATH, suggesting that successfully solving the question in MATH$^2$ requires a nontrivial combination of two distinct math skills.

1. Introduction

The introduction argues that established sources of challenging math questions are becoming scarce, while human-only creation is not scalable and LLM-generated questions often lack difficulty or diversity. It proposes combining LLM skill-pair generation with human review to create more novel, challenging questions.

1. Introduction

  • High-quality, varied, difficult math questions are becoming scarce, while relying only on experts is not scalable and LLM-generated questions often lack difficulty or diversity.The authors also note that related synthetic-question efforts have largely targeted model fine-tuning rather than evaluation.Source: §1 · pp. 1–2

1.1. Evaluation Saturation Phenomenon

  • Synthetic training can boost benchmark scores, but performance variation across test-set versions raises concerns about overfitting rather than genuine mathematical skill.The cited example reports that one million synthetic examples can bring Llama2 7B's MATH performance to levels comparable to GPT-4.Source: §1.1 · p. 2

1.2. Proposed Framework: AI-assisted Generation of Difficult Math Questions

  • The framework extracts skills from MATH, pairs skills from different areas, and uses LLMs and human experts to generate, select, and refine questions.Experts focused on promising questions, including ones the LLM answered incorrectly despite containing correct ideas, then refined them and supplied gold-standard answers.Source: §1.2 · p. 2
  • Combining skills across subareas is intended to prompt out-of-distribution thinking, and the authors report that MATH2 challenges both LLMs and human raters.The authors suggest the framework may also inform math instruction and benchmark generation in areas such as code and formal math reasoning.Source: §1.2 · pp. 2–3
  • The authors connect novel human-AI evaluation to scalable oversight: experts may need new ways to assess models trained on available exams and textbooks.They pose human-AI collaboration as a possible route to more novel evaluations.Source: §1.2 · p. 3

2. Pipeline for AI-Assisted Question Generation

The pipeline extracts mathematical skills, generates questions requiring distinct skill pairs, and iteratively checks questions and solutions. Human screening provides a further quality check.

2. Pipeline for AI-Assisted Question Generation

  • The pipeline starts by extracting and cataloging mathematical skills linked to dataset questions, then samples skill pairs and reference question-solution examples from MATH.Pairs judged too similar are excluded before question generation.Source: §2 · pp. 3–4
  • The LLM generates a question and brief solution requiring both sampled skills, with prompts specifying a single final result and exact or explicitly approximate answers.In-context human-AI exchanges demonstrate feedback and revision to help avoid insufficient skill use and logical inconsistencies.Source: §2 · p. 4
  • The model first attempts an adversarial solution to identify flaws, then LLM validation checks accepted questions against a seven-criterion rubric using majority voting.The attempted solution targets issues including ambiguity, contradiction, insufficient information, and excessive computation.Source: §2 · p. 4
  • Questions classified as valid are re-solved with in-context examples and majority voting, and questions with unique answers across attempts are discarded as potentially ambiguous.Humans further screen the resulting questions to help ensure quality.Source: §2 · pp. 4–5

3. Experiments and Findings

The experiments compare MATH2 with MATH and test MATH2 questions as skill-matched in-context examples. Models score lower on MATH2, while MATH2 exemplars improve MATH performance over two baselines.

3.1. Experimental Setup

  • MATH2 contains 210 verified difficult questions; human annotators modified 130 (61.9%) question-solution pairs and 97 originally generated solutions, while 38.1% of question-answer pairs remained exactly as phrased by LLM creators.Source: §3.1 · p. 5

3.2. Performance across the two datasets: A surprising pattern

  • All tested models score lower on MATH2 than on MATH; frontier-model percentage drops range from 10.89% to 42.42%, while MetaMath-13B drops 97.33%.Source: Table 1 · p. 5, §3.2 · p. 6
  • Across models, MATH2 performance follows Y ≈ X^2 against MATH performance; a heuristic random-pair argument and the authors’ interpretation point to questions requiring two distinct skills.Source: §3.2 · p. 6
  • Three of five recent thinking models deviate positively; DeepSeek-R1-Distill-Llama-8B is ∼10.5% above Y = X^2, while Claude-3.5 Sonnet, MetaMath, and MAmmoTH show the largest negative deviations.Source: §3.2 · p. 6

3.3. Generated Questions are Effective In-Context Exemplars for MATH.

  • For 4-shot CoT on filtered MATH questions, skill-matched MATH2 exemplars outperform MAmmoTH and skill-based baselines; 364 test questions were removed for unsupported skills.Source: §3.3 · p. 7

4. Observations from the Question Generation Process

The generation process combines iterative LLM refinement with human verification and editing. Examples show humans adapting creative questions to increase challenge or improve tractability, while models still produce recurring failure modes.

4.1. Creative questions: Examples of Synergy from Human-AI interaction

  • Models often produced creative questions but incorrect solutions that contained enough correct ideas for a human to complete quickly.Source: §4.1 · p. 7
  • Human raters verified question validity and solution correctness, checked intended skills, and improved readability, quality, and difficulty during the iterative generation process.Source: §4.1 · p. 7, §4 · p. 7
  • For a bead-sequence problem, the human changed the sequence size from 20 to 6, preserving essential difficulty while making it more elegant for people.Source: §4.1 · pp. 7–8
  • Up to 13.72% gain over baseline (Llama-3.1-70B-Instruct) came from MATH2 exemplars among the three MATH prompting strategies compared.Source: Table 2 · p. 8
  • For a water-growth problem, the human simplified hand calculations and replaced a condition with one models found challenging but humans could estimate and verify.Source: §4.1 · p. 8
  • A recursive-sequence question easy to solve by computation was modified to require recognizing an underlying pattern.Source: §4.1 · p. 8
  • Rephrasing a prime-divisor problem made direct computation impractical and required modular arithmetic and prime factorization, illustrating the role of human refinement.Source: §4.1 · p. 8
  • The pipeline still frequently produces questions with insufficient skill use or information, unsolvable or intractable problems, nonsensical questions, and deceitful solutions.Source: §4.1 · p. 8

5. Conclusions

The pipeline combines LLM skill extraction and paired-skill question generation with iterative refinement and human review, producing questions that are more novel and difficult than MATH. MATH² questions also improve model performance on MATH when used as in-context examples.

5. Conclusions

  • The pipeline extracts and names skills from MATH, prompts advanced LLMs to combine skill pairs through multiturn interactions, then has human raters review and refine the questions.The resulting questions are described as more novel and difficult than those in the original MATH dataset.Source: §5 · p. 8
  • MATH² questions are significantly more challenging than MATH while assessing the same skills, and using them as in-context examples improves performance on MATH.The questions combine two skills from different parts of MATH; o1-preview and DeepSeek-R1 distilled models showed the smallest performance drops.Source: §5 · p. 8
  • The pipeline incurs moderately high costs from extensive frontier-model API use and significant human verification.Future work is proposed on open-weight models, prompting efficiency, automated validation, and feedback from questions that pass human verification.Source: §5 · p. 8
  • The framework may also be applied to producing high-quality data in domains beyond mathematics.The passage presents this as a prospective application.Source: §5 · p. 9

A.1. Validation Criteration for the Question Validation Step

The validation step filters generated questions using seven criteria covering answer uniqueness, use of both skills, clarity, tractability, realism, and writing quality.

A.1. Validation Criteration for the Question Validation Step

  • Questions must request one final answer and have one exact answer unless approximations are explicitly allowed.Source: §A.1 · p. 12
  • Both skills must be necessary and sufficient, with difficulty comparable to or greater than the reference examples.Source: §A.1 · p. 12
  • Questions must be clear, complete, and solvable without overly complex computations.Source: §A.1 · p. 12
  • Scenarios must be realistic and logically consistent, and questions must be grammatically correct and clearly written.Source: §A.1 · p. 12

A.2. Statistics of Human Verification in MATH2

Human annotators modified at least one question or solution in 130 of the 210 MATH² examples, often to increase difficulty or correct errors.

A.2. Statistics of Human Verification in MATH2

  • 130 of 210 MATH² examples (69.1%) had at least one model-generated question or solution modified by an annotator.Source: Table 3 · p. 12
  • Annotator modifications aimed to increase question difficulty or correct questions or solutions.Source: Table 3 · p. 12

A.3. Failure Modes and Interesting Behaviors

Generated questions can fail to use both skills adequately, omit necessary information, be computationally impractical or logically flawed, and survive automated checks, so human verification remains necessary.

Insufficient involvement of skills

  • Insufficient involvement of one or both skills is the most prominent failure mode and can yield easy questions; multi-turn prompting and stronger validation nearly eliminate missing skills and reduce shallow use.Source: §A.3 · p. 13

Insufficient information

  • Questions may omit essential details, and attempting a solution can help reveal missing information not apparent from the question alone.Source: §A.3 · p. 13

Unsolvable or computationally intractable questions

  • Some questions are unsolvable or demand tedious brute-force calculations; the pipeline adds solvability and tractability checks assisted by model solution attempts.Source: §A.3 · pp. 13–14

Nonsensical questions

  • Generated questions can be nonsensical or ambiguous, including a square-inside-quarter-circle scenario that is geometrically inconsistent.Source: §A.3 · p. 14

Deceitful solutions

  • Models sometimes fabricate plausible solutions to unsolvable questions; validation and consistency checks screen most such cases but do not eliminate them.Source: §A.3 · pp. 13–14

Finding a needle in the haystack

  • Long validation prompts can make models overlook the target skill pair or substitute unrelated skills; repeating the skill names nearly eliminated this problem.Source: §A.3 · pp. 14–15

Human verification

  • The described failure modes are not completely eliminated by the pipeline, so human verification is required.Source: §A.3 · p. 16

Interesting behaviors

  • In one case, a model flagged brute-force burden as a concern without an explicit complexity check, which the authors attribute to role-playing prompts.Source: §A.3 · p. 16

A.4. Considerations for human-annotaters

Human annotators checked question validity and solution correctness, then refined readability, quality, and difficulty. Examples show edits intended to move questions beyond simple computation or brute-force enumeration.

A.4. Considerations for human-annotaters

  • Annotators checked that questions used their intended skills and improved readability, quality, and difficulty while seeking problems harder for automated tools but easy for humans.They also checked solution correctness and watched for the failure modes discussed in Section 4.Source: §A.4 · p. 16
  • A recursive-sequence question that an LLM could solve by simple computation was modified to require understanding the underlying pattern.For the modified question, one leading model could not answer and another gave an incorrect answer after an incorrect numerical approximation.Source: §A.4 · pp. 16–17
  • The 210 generated and human-verified questions cover 109 of 114 extracted skills, with two skills represented in each question and a nonuniform skill distribution.Number theory and perimeter-and-area skills occur most frequently; several skills appear in only one question.Source: Figure 4 · p. 17
  • Annotators revised questions involving tedious enumeration so that brute-force computation would not be a feasible strategy.The prime-divisor example was changed from finding the smallest and largest prime divisors of 154 + 164 to finding the two smallest prime divisors of 2317 + 1717.Source: §A.4 · p. 17
  • After rephrasing a prime-divisor question, models had to reason more analytically, including understanding arithmetic modulo a prime, rather than use brute force.The original question invited calculating 154 + 164; the modified question asks for the two smallest prime divisors of 2317 + 1717.Source: §A.4 · p. 17

A.5. Further Experimental Details and Results

The reported experimental setup specifies the hardware and parallel workers used for inference across open and proprietary models.

A.5. Further Experimental Details and Results

  • Open-source model inference used four 80GB A100 GPUs and 72GB of RAM, facilitated by vLLM.The cited tool is vLLM (Kwon et al., 2023).Source: §A.5 · p. 17
  • The authors used 50 parallel workers for o1-preview, GPT-4 Omni, GPT-4 Turbo, and Gemini-1.5-Pro.Source: §A.5 · p. 17
  • They used two parallel workers for Claude-3 Opus and Claude-3.5-Sonnet.Source: §A.5 · p. 17

A.5.1. EFFICIENCY AND COST OF THE QUESTION GENERATION PIPELINE

The section reports how questions are filtered by the AI pipeline and human verification, and estimates per-model generation costs. GPT-4 Turbo is identified as the most efficient at getting generated questions through the AI pipeline.

A.5.1. EFFICIENCY AND COST OF THE QUESTION GENERATION PIPELINE

  • Table 4's representative pipeline counts track generation, validation rejections, unique final answers across four solution traces, and parsing errors.The reported batches are representative; questions in MATH2 do not necessarily belong to those batches.Source: §A.5.1 · pp. 17–18, Table 4 · p. 18
  • GPT-4 Turbo is the most efficient model by the number of originally generated questions that make it through the AI pipeline.Table 4 reports questions generated and rejected at validation, majority agreement, and parsing stages.Source: Table 4 · p. 18, §A.5.1 · p. 18
  • Human annotators classified questions as good, a bit too easy, or wrong after possible modifications; only questions judged good entered the dataset.Their judgments combined qualitative assessment with techniques such as checking how frontier models approached a question.Source: Table 5 · p. 18, §A.5.1 · p. 18
  • The estimated total generation costs are an upper bound because questions rejected during validation can avoid solution-generation costs.The cost estimate uses average prompt and generation lengths over 20 interactions; the table caption describes the estimate as pessimistic.Source: §A.5.1 · p. 18, Table 6 · p. 19

A.5.2. SKILL PROPORTIONAL COMPARISON OF MATH2 AND MATH

To isolate the effect of combining skills, the authors compare MATH2 with a randomly sampled MATH subset matched to its skill distribution. They conclude that the matched subset is easier than both MATH2 and MATH.

A.5.2. SKILL PROPORTIONAL COMPARISON OF MATH2 AND MATH

  • The authors conclude that a 2,983-question MATH subset with a skill distribution similar to MATH2 is easier than both MATH2 and MATH.They formed the subset by randomly sampling MATH questions by skill; Table 7 compares model performance across the three sets.Source: §A.5.2 · p. 19, Table 7 · p. 19

A.5.3. DIFFICULTY OF QUESTIONS GENERATED BY DIFFERENT MODELS

This section describes how questions combining two mathematical skills are generated, checked, solved, and evaluated, including the prompts and illustrative examples used in the pipeline. It also reports how question difficulty varies by generating model and human modification.

A.5.3. DIFFICULTY OF QUESTIONS GENERATED BY DIFFERENT MODELS

  • Gemini-1.5-Pro-generated questions were most difficult, followed by GPT-4 Turbo and Claude-3 Opus, and questions were not necessarily easier for their generating model.Source: Table 8 · p. 19, §A.5.3 · p. 19

A.5.4. MODIFIED QUESTIONS VS NON-MODIFIED QUESTIONS

  • Human-modified MATH2 questions were more difficult than MATH Level-5 questions; annotators corrected errors or unclear wording and could make questions more difficult.Source: Table 9 · p. 20, §A.5.4 · p. 20

A.6. Skills used for generating the questions

A.7. Example outputs for each step of the AI pipeline

  • A worked pipeline example combines sequence_analysis and polynomial_operations in a sequence question asking for the units digit of a product.The example’s validation describes the question as requiring both skills and as having a single exact answer.Source: §A.7 · pp. 21, 23

A.8. Skill composition using different models

  • The appendix presents responses from five models prompted to combine area-calculation and algebraic-manipulation skills, including examples about a triangle within a rectangle and an equilateral-triangle-and-squares construction.Source: §A.8 · pp. 23, 25
  • Illustrative two-skill questions combine area calculation with algebraic manipulation, including circle and semicircle geometry and integration of a curve.The examples describe nested radicals and expressions involving π as added complexity.Source: §A.8 · pp. 26–27

A.9. Prompt Examples

  • The appendix presents prompt skeletons with placeholders for skill names and exemplars, while making the in-context exemplars and skill-exemplar repository available.Source: §A.9 · p. 27

A.9.1. SKILL PAIR VALIDATION

  • Skill-pair validation asks whether two skills are essentially the same, using their descriptions and sample question-solution pairs as context.The requested response explains its reasoning and ends with a yes-or-no judgment about similarity.Source: §A.9.1 · pp. 27–28

A.9.2. QUESTION GENERATION

  • Question generation requests one non-multiple-choice problem requiring both skills, a brief solution, and self-review, with exemplar-based difficulty and a single exact result unless approximation is explicit.The prompt also asks that the application difficulty of each skill match the difficulty shown in its examples.Source: §A.9.2 · p. 28

A.9.3. ATTEMPTED SOLUTION

  • The attempted-solution prompt directs the model to stop and explain if a question is flawed, ambiguous, intractable, or requires unspecified approximation.If no such issue arises, the model should solve for the single exact answer; the prompt is described as taking a defeatist approach.Source: §A.9.3 · p. 29

A.9.4. QUESTION VALIDATION

  • Question validation checks for a single exact answer, essential use of both skills, adequate clarity, and hand-solvable computations under exam conditions.The prompt also screens for relevance, realistic and logically consistent setups, and grammatical clarity.Source: §A.9.4 · pp. 29–30

A.9.5. FINAL SOLUTION

  • The final-solution prompt uses MATH in-context exemplars, unlike the attempted-solution step.Source: §A.9.5 · p. 31
  • The final-solution prompt requires each new question to compose both skills, then requests a step-by-step solution and a final numerical answer.The answer is required in a specified format with the explanation and numerical answer on separate lines.Source: §A.9.5 · pp. 31–32

A.9.6. EVALUATION

  • GPT-4 evaluates whether a student’s reasoning is technically correct, marking a matching final answer incorrect when the reasoning is wrong.The solution approach may differ from the ground truth, and the evaluation ends with a Yes or No correctness judgment.Source: §A.9.6 · p. 32
Loading 2407.21009v4…