PaperlayerView exact revision on arXiv

Source-linked AI summary

HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics

Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, Michael P. Brenner

arXiv:2410.09988v2cs.LGcs.AI

TL;DR

Existing LLM mathematics benchmarks underrepresent applied problems requiring approximate analytical solutions. HARDMATH automatically generates challenging problems and evaluates models on test sets; models remain less accurate on HARDMATH-MINI than on existing mathematics benchmarks.

  • Problem

    Applied problems requiring approximate analytical solutions are underrepresented in LLM benchmarks, despite their prevalence in science and engineering.

    Source: §1 · pp. 1–2

  • Method

    HARDMATH algorithmically generates problems and step-by-step solutions, then evaluates models on HARDMATH-MINI and applied-science word problems.

    Source: §1 · pp. 1–2

  • Results

    Across HARDMATH-MINI, few-shot CoT improves performance, but accuracy remains much lower than on existing benchmarks; GPT-4 scores 43.8% overall with 5-shot CoT.

    Source: §4.3 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    HARDMATH offers a benchmark for evaluating LLMs’ ability to make research-relevant approximations.

    Source: §1 · p. 1

    The reported evaluation uses HARDMATH-MINI; the authors plan to fine-tune on the larger HARDMATH and evaluate more models.

    Source: §5 · p. 10

Abstract

from arXiv · show

Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs.

1 Introduction

The introduction motivates HARDMATH as a benchmark for applied mathematical approximation, a mode of reasoning underrepresented in existing LLM evaluations. It presents an automatically generated dataset and reports that current models perform poorly on its challenging problems.

1 Introduction

  • The benchmark targets approximate analytical solutions to real-world problems that often lack closed-form solutions, beyond the focus of many existing mathematical reasoning datasets.Numerical solutions can offer insight but may not provide intuition about solution behavior.Source: §1 · p. 1
  • HARDMATH contains 1,466 problems inspired by a graduate-level asymptotic methods course, covering algebraic equations, ODEs, and integrals.The problems reflect scientific and engineering contexts where exact solutions often do not exist.Source: §1 · pp. 1–2
  • The paper identifies a shortage of benchmarks for approximation methods used in applications, despite their difficulty and prevalence in science and engineering.These problems can require advanced calculus, differential equations, complex analysis, computational tools, and numerical benchmarks.Source: §1 · p. 2
  • Current LLMs perform poorly on HARDMATH’s challenging asymptotics problems, according to the introduction’s evaluation summary.The paper describes its evaluation accuracy summary and error-mode analyses.Source: §1 · p. 2
  • Algorithms automatically generate problems and step-by-step solutions, while HARDMATH, HARDMATH-MINI, and WORD-PROBLEMS-HARDMATH support development and evaluation.The paper reports an evaluation accuracy summary and error-mode analyses.Source: §1 · p. 2

2 Related work

The related-work discussion distinguishes HARDMATH from datasets centered on simpler problems or manually sourced formal mathematics. It places the benchmark amid growing interest in harder mathematical evaluations as frontier models advance.

2.1 Mathematical datasets

  • Existing advanced datasets are often small, difficult to scale, focused on abstract formal mathematics, or drawn from copyrighted textbooks, while HARDMATH targets applied mathematics.The surveyed examples include JEEBENCH [Arora et al., 2023], MATHBENCH [Liu et al., 2024], GHOSTS [Frieder et al., 2024], and ARB [Sawada et al., 2023].Source: §2.1 · p. 2
  • HARDMATH’s generator can produce additional problems, addressing the scale and focus limitations the authors identify in existing datasets.The paper describes HARDMATH as a large collection of challenging applied mathematics problems inspired by a graduate-level asymptotic methods course.Source: §2.1 · p. 2
  • The authors describe HARDMATH as the largest graduate-level dataset and emphasize its integration of tools with sophisticated reasoning.The cited discussion says success requires combining tool use and sophisticated reasoning, distinguishing HARDMATH from other mathematical datasets.Source: §2.1 · p. 3

2.2 Recent interest in advanced mathematical reasoning

  • Interest in harder mathematics benchmarks is growing as frontier models advance and existing benchmarks may no longer differentiate them [OpenAI, 2024].The discussion also cites Humanity’s Last Exam as an open challenge seeking questions requiring PhD-level academic training.Source: §2.2 · p. 3

3 Datasets

HARDMATH combines asymptotic-method problems with algorithmically generated solutions checked against numerical results. Its problem types cover polynomial methods, nonlinear ODEs, integrals, and word problems, including automatically generated scientific contexts.

3.1 HARDMATH design choices

  • HARDMATH problems use dominant balance alongside self-consistency checks and numerical methods, with justified choices of regimes and approximation detail.These choices combine analytical and computational techniques.Source: §3.1 · p. 4

3.2 Dataset generation and verification

  • Generated problems receive step-by-step solutions, and inclusion requires under 10% error against numerical ground truths; HARDMATH-MINI solutions are also visually checked against numerical results.Polynomial root corrections are checked to improve the original approximation.Source: §3.2 · pp. 4–5

3.3.1 Nondimensionalization of polynomials

  • Polynomial nondimensionalization recasts equations into scaled forms, including ϵy^n1 + y^n2 + 1.The symbolic form begins with a1x^n1 + a2x^n2 + a3, where n1 > n2 > 0.Source: §3.3.1 · p. 5

3.3.2 Polynomial root-finding; 3.3.3 Polynomial root correction terms

  • Higher-order polynomial roots use dominant balance for small and large positive ϵ, while a Taylor-derived correction δ can reduce approximation error.The correction is added to the approximate root and derived by expanding around that approximation.Source: §3.3.2 · p. 5, §3.3.3 · p. 5

3.3.4 Nonlinear ordinary differential equations

  • The nonlinear ODE problems provide approximate formulae for small x near zero and large x, typically where the solution diverges; the dataset includes third-order ODEs.Source: §3.3.4 · p. 5

3.3.5 Traditional integrals; 3.3.6 Laplace integrals

  • Traditional integral problems use approximations for small, intermediate, and large ϵ, while Laplace integrals use Laplace’s Method for large x when values concentrate near t0.Source: §3.3.5 · p. 5, §3.3.6 · p. 6

3.4 Word problems in context; 3.5 Automatic generation of context for word problems

  • The 40 manually generated word problems test contextual effects, while an LLM-generated physics set is screened for plausibility and spans physics subfields.Source: §3.4 · p. 6, §3.5 · p. 6

3.5 Automatic generation of context for word problems

  • Most automatically generated contexts scored plausibility >0.5, and the problems span different subfields of physics.The context-generation approach is described as limited in capturing subtler domain-specific priors.Source: §3.5 · p. 6

4 Evaluation

The evaluation combines answer checking with procedural grading, then compares open- and closed-source models on HARDMATH-MINI and contextual word problems. Few-shot CoT improves performance, but accuracy remains limited; fine-grained analysis shows different response and error patterns across problem types.

4.1 Evaluation protocols

  • 366 HARDMATH-MINI problems cover four types, and evaluation combines answer-equivalence checks with procedural grading for Roots, ODEs, and Integrals.GPT-4o serves as the procedural grader, and the authors manually verify a subset of its grading responses.Source: §4.1 · pp. 6–7

4.2 Model choice

  • The comparison includes closed-source GPT-3.5, GPT-4, and o1-mini, plus open-source Llama3 and CodeLlama, in zero- and few-shot CoT settings.The paper describes this model subset as representative of current LLM capabilities.Source: §4.2 · p. 7

4.3 Quantitative results

  • 62.3% overall accuracy is achieved by o1-mini with 5-shot CoT; GPT-4 scores 43.8% and Llama3-8b 20.2% with 5-shot CoT.The reported results cover the HARDMATH evaluation set.Source: §4.3 · p. 7, Table 2 · p. 8

4.3.1 Extensions to word problems

  • 28.1% overall accuracy is reported for GPT-4 on 40 hand-crafted science and engineering word problems without problem-specific hints.The set mixes Nondim, Roots, ODEs, and Integrals problems.Source: §4.3.1 · p. 8

4.4 Fine-grained results

  • For harder ODE problems, full correctness is rare and CoT increases partial-credit responses; on simpler Roots problems, advanced models produce more fully correct answers as shot count rises.The analysis attributes partial responses on ODEs to models addressing easier small x regime solutions without solving the problems entirely.Source: §4.4 · p. 8, Figure 3 · p. 9
  • In GPT-4 Roots answers, 5-shot CoT shifts errors from leading-term-only dominant balances toward missed cases and complex-root calculations.o1-mini sometimes returns only final answers, making its error sources difficult to trace.Source: §4.4 · pp. 8–9, Figure 4 · p. 9

3. Grading Response: Roots

The example grades a response that identifies only one of the required root balances as partially correct.

3. Grading Response: Roots

  • The response includes roots from balance B + C = 0 but misses A + B = 0, earning 0.5 for small positive ϵ.The prompt asks for approximate roots in the small- and large-positive-ϵ limits.Source: §3 · p. 9

5 Conclusion

HARDMATH provides generated applied-mathematics problems and human-verified evaluation sets. The evaluation finds that few-shot CoT improves performance, while accuracy remains below existing benchmarks; the paper identifies evaluation on HARDMATH-MINI as its current scope.

5 Conclusion

  • HARDMATH contains 1,060 generated problems, alongside 366 human-verified HARDMATH-MINI examples and 40 human-verified contextual problems.The larger dataset is intended for model development, including fine-tuning.Source: §5 · p. 10
  • Few-shot CoT significantly improves performance, but HARDMATH-MINI accuracy remains much lower than on existing benchmarks.The authors conclude that these tasks remain challenging for current LLMs, particularly in approximation-oriented mathematical reasoning.Source: §5 · p. 10
  • The reported evaluation uses HARDMATH-MINI; fine-tuning on the larger HARDMATH dataset is left to future work.The authors also plan to extend evaluations to additional frontier models as they become available.Source: §5 · p. 10

A.1 Semi-automated solution verification

Figure 5 compares numerical and approximate analytical solutions for a sample Laplace integral problem. The comparison supports semi-automated human verification against numerical ground truths.

A.1 Semi-automated solution verification

  • The comparison supports semi-automated human verification that analytical solutions correspond well with numerical ground truths.This verification method was used for HARDMATH-MINI.Source: §A.1 · p. 13
  • Figure 5 compares numerical and approximate analytical solutions for a sample Laplace integral problem.The comparison spans a large domain.Source: Figure 5 · p. 13, §A.1 · p. 13

A.2 Implementation and method details for data generation

The following subsections describe how problems and solutions are generated for each problem type.

A.2 Implementation and method details for data generation

  • The following subsections detail the process used to generate problems and solutions for each problem type.The passage introduces the generation-process descriptions that follow.Source: §A.2 · p. 13

A.2.1 Nondimensionalization of polynomials

The polynomial nondimensionalization problems vary polynomial degrees and coefficient settings, then express the dimensionless parameter ϵ in terms of coefficients or as a numerical value.

A.2.1 Nondimensionalization of polynomials

  • Degrees n1 and n2 vary with 0 < n2 < n1 < 10 while symbolic positive coefficients yield ϵ as a function of a1, a2, and a3.The symbolic subtype specifies a generalized first nondimensionalization problem.Source: §A.2.1 · p. 13
  • The symbolic sample asks for ϵ in a polynomial of the form ϵy10 + y9 + 1, expressed as a function of its coefficients.The example begins with a1x10 + a2x9 + a3 and uses a substitution before dividing by the remaining constant-term coefficient.Source: §A.2.1 · p. 13
  • A second subtype uses integer coefficients a1, a2, and a3 randomly chosen from [−10, 10].A sample numeric problem nondimensionalizes a polynomial into the form ϵy7 ± y2 ± 1.Source: §A.2.1 · p. 14
  • The numeric sample substitutes coefficient values into the nondimensionalized polynomial and reports ϵ ≈ 0.08.The displayed numeric result includes 1024, y2, and 1.Source: §A.2.1 · p. 14

A.2.2 Polynomial root-finding

For polynomial root-finding, the method uses dominant balances and checks their consistency across small- and large-ϵ limits. The example reports a complete set of six roots in each regime.

A.2.2 Polynomial root-finding

  • The generated root-finding polynomials have maximum order ten, with degrees satisfying 0 < n2 < n1.The sample asks for first-order approximations in both small- and large-positive-ϵ limits.Source: §A.2.2 · p. 14
  • Roots from each dominant balance are checked by substituting them into A, B, and C and comparing term magnitudes.The problem is rewritten as A + B + C = 0, and the approximate roots are tested for small or large positive ϵ validity.Source: §A.2.2 · pp. 14–15
  • The method finds six roots valid in the small-positive-ϵ limit and six valid in the large-positive-ϵ limit, matching the degree-six polynomial's root count in each regime.The passage describes the solution as complete.Source: §A.2.2 · p. 15

A.2.3 Polynomial root correction terms

The appendix presents asymptotic-method applications and evaluation materials, spanning polynomial-root corrections, ODE and integral approximations, applied-science word-problem generation, and examples of response grading. It also examines failures in root counting and integral approximation, a nondimensionalized quintic, and consistent balances across parameter regimes.

A.2.3 Polynomial root correction terms

  • Polynomial-root corrections use the ansatz x + δ, expand in δ, discard higher-order terms, and solve for δ to O(1).The correction represents higher-order terms neglected in the initial approximation.Source: §A.2.3 · p. 16

A.2.4 ODEs

  • The generated third-order ODEs use rational-function terms and integer initial conditions; solutions use a third-order Taylor expansion near zero and dominant balance for large x, assuming power-law divergence.Problems with a function of x as the dominant term are excluded.Source: §A.2.4 · p. 19

A.2.5 Integrals

  • Generated polynomial integrals avoid oscillation; their approximation regimes use the lowest-degree term for small ϵ, the highest-degree term for intermediate ϵ, and L/ϵ when the width exceeds the integration range.The polynomials have up to ten positive-power terms, and the integration bound is randomly selected.Source: §A.2.5 · p. 20

A.2.6 Laplace integrals

  • Laplace-integral problems vary their bounds, functions, and exponent sign; the worked example approximates the large-x integral near the maximum of f(t), located at t = −0.66.The example finds the maximum by evaluating critical points and interval bounds; SymPy is used to find minima, with dual annealing as a fallback.Source: §A.2.6 · pp. 20–21

A.2.8 Word problem automatic generation example

  • A word-problem example recasts a third-order ODE approximation task as fluid flow through a porous medium, and the quality figure includes only problems above a plausibility threshold.Source: §A.2.8 · pp. 22–23, Figure 6 · p. 23

A.4.1 Extended evaluation results

  • CoT accuracy increases with shot count across models, with the clearest improvements for o1-mini and GPT-4; Figure 7 also breaks down 5-shot accuracy by problem subtype.The supplied passage gives no numerical accuracy values.Source: Figure 7 · p. 27

A.4.2 Extended error mode analysis

  • A model’s root analysis fails to find nonzero roots for large ϵ, while the reference balances ϵx^6 against −1 and verifies validity for large, but not small, ϵ.The example identifies incorrect dominant-balance setup as the error mode.Source: §A.4.2 · p. 27
  • For large positive ϵ, one polynomial response earned 0.5 for omitting two complex roots, while another earned 0.0 for an invalid balance and no concrete root list.The reference solution identifies three nonzero roots from the dominant balance ϵx^3 − 1 = 0 and validates that balance only for large ϵ.Source: §A.4.2 · pp. 28–29

A.4.3 Extended example GPT-4o grading responses

  • GPT-4o’s ODE grading compares the response with ground truth in both small x and large x regimes.The example then finds the small-x response wrong because its exponential term differs from the ground-truth polynomial.Source: §A.4.3 · p. 29, Small x Regime · p. 29

Small x Regime

  • In the small x regime, the response’s exponential term differed from the ground-truth polynomial, so the solution was judged completely wrong.The comparison found that the two forms did not match.Source: Small x Regime · p. 29

Large x Regime

  • For the large x ODE regime, a response that asserted rapid growth without a specific analytical expression did not match the ground truth and received no credit.The grading judged both the small- and large-regime solutions incorrect or incomplete.Source: Large x Regime · p. 29
  • For integral examples, matching the ground truth in both regimes earned full credit 1.0, while a heuristic Laplace treatment with an inaccurate coefficient earned 0.5.The partial-credit response recognized the dominant exponential behavior and significant point but omitted the detailed Taylor expansion and change of variables used by the answer key.Source: Large x Regime · p. 30

A.5 Motivation for including nondimensionalization problems

  • The quintic-root example nondimensionalizes to ϵx^5 − x + 1 = 0 and uses self-consistency checks to distinguish valid balances in the ϵ → 0 limit.The reference solution rejects the ϵx^5 ∼ −1 balance and accepts ϵx^5 ∼ x as self-consistent.Source: §A.5 · p. 32
Loading 2410.09988v2…