PaperlayerView exact revision on arXiv

Source-linked AI summary

U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs

Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, Sergei Tilga

arXiv:2412.03205v4cs.CLcs.AI

TL;DR

Existing benchmarks provide limited coverage of university-level topics, visual reasoning, and the evaluation of free-form mathematical solutions. U-MATH and µ-MATH provide university-level problem and judge-evaluation benchmarks; experiments find visual-task performance below text-only performance and solution judgment remains challenging.

  • Problem

    Existing math benchmarks are limited in size, topic diversity, university-level coverage, and visual reasoning, while LLM judges of free-form solutions can be biased and inconsistent.

    Source: §1 · p. 1

  • Method

    The paper constructs U-MATH from university coursework and derives µ-MATH, a dataset of model-generated solutions with correctness labels for evaluating LLM judges.

    Source: §1 · p. 1, §3.1 · p. 4, §5 · p. 7

  • Results

    Visual-task accuracy is 58.5% versus 93.1% for text-only tasks, while solution judges peak at an imperfect 90.1% F1-score.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Judgment is distinct from problem-solving, and the observed judging biases and instability underscore the utility and necessity of meta-evaluations.

    Source: §1 · p. 2, §5 · p. 7

    Visual problems are not covered by µ-MATH, and the 20% visual share limits the scope of visual-reasoning evaluation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The current evaluation of mathematical skills in LLMs is limited, as existing benchmarks are either relatively small, primarily focus on elementary and high-school problems, or lack diversity in topics. Additionally, the inclusion of visual elements in tasks remains largely under-explored. To address these gaps, we introduce U-MATH, a novel benchmark of 1,100 unpublished open-ended university-level problems sourced from teaching materials. It is balanced across six core subjects, with 20% of multimodal problems. Given the open-ended nature of U-MATH problems, we employ an LLM to judge the correctness of generated solutions. To this end, we release $\mu$-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions. Benchmarking leading LLMs reveals marked limitations in multi-modal reasoning, with maximum accuracy reaching 93.1\% on textual tasks but only 58.5\% on visual ones. Furthermore, solution judgment proves challenging, requiring the most advanced models to achieve meaningfully high performance, even still peaking at an imperfect F1-score of 90.1\%.

1 Introduction

Existing benchmarks leave gaps in university-level topic coverage, visual reasoning, and evaluation of free-form solutions. U-MATH and µ-MATH address these gaps, and comparative analysis finds persistent weaknesses in multimodal reasoning and solution judgment.

1 Introduction

  • Existing benchmarks primarily cover school-level problems or offer limited scope and competition-style tasks, while visual benchmarks may restrict topic coverage or use easier multiple-choice formats.Evaluating visual reasoning and reliably evaluating free-form responses are further identified as open needs.Source: §1 · p. 1
  • U-MATH contributes 1,100 open-source problems across six core university subjects, collected from actual coursework, with approximately 20% containing visual elements.The problems are supplied with correct answers.Source: §1 · p. 1
  • µ-MATH contributes 1,084 meta-evaluation tasks using four model-generated solutions for approximately 25% of U-MATH problems, with ground-truth correctness labels.These tasks are designed to assess the quality of LLM judges.Source: §1 · pp. 1–2
  • LLMs show significant deficiencies on university-level multimodal problems, while solution judgment remains challenging and biased or unstable for many systems.The analysis also finds that judgment is distinct from problem-solving and that models exhibit characteristic judging behaviors.Source: §1 · p. 2

2 Background

Prior math benchmarks often lack the size, topic breadth, university-level coursework focus, or open-ended format needed for broad assessment. Existing mathematical meta-evaluation datasets also largely inherit the limitations of GSM8K and MATH.

2 Background

  • Many textual benchmarks cover elementary- or high-school-level material, approach saturation, or emphasize competition problems rather than university coursework.Early multiple-choice benchmarks have also become obsolete as smaller models achieve high scores.Source: §2 · p. 2
  • U-MATH offers over three times as many open-ended university-level problems as the cited alternatives, sourced from previously unpublished materials rather than public datasets.Many alternatives have fewer than 400 problems each and focus on Olympiad-style topics rather than practical university coursework.Source: §2 · p. 3
  • Visual math benchmarks range from narrow geometry coverage to college-level collections whose multiple-choice format can limit task complexity, while other visual sets have topic or format limitations.U-MATHVisual uses free-form responses for visual problems while adhering to university-course topics.Source: §2 · p. 3
  • Existing mathematical meta-evaluation datasets are primarily based on GSM8K and MATH, leaving advanced mathematical areas underrepresented; µ-MATH instead uses U-MATH problems for university-level meta-evaluations.Meta-evaluation matters because free-form LLM judging is prone to errors and biases, and judgment performance does not fully align with problem-solving performance.Source: §2 · p. 3

3 U-MATH

U-MATH is a benchmark of 1,100 university-level problems, curated from coursework to emphasize reasoning and spanning six subjects with text and visual items. Its companion µ-MATH provides a selected, challenging set for evaluating LLM judgments of mathematical solutions.

3 U-MATH

  • U-MATH contains 1,100 problems for evaluating university-level mathematics, and a problem counts as solved only when every included question is answered correctly and fully.Accuracy is the main metric, with an LLM judge comparing responses against golden labels.Source: §3 · p. 4

3.1 Dataset Curation

  • Problems and solutions came from ongoing university courses, and curation selected challenging items while filtering short, multiple-choice, and calculator-implying tasks.Experts reviewed problems and golden labels; academic reviewers judged university-level suitability, with 4.3% categorized as high-school rather than university-level.Source: §3.1 · p. 4

3.2 Dataset Statistics

  • The benchmark spans six subjects and 1,100 problems, with about 20% including visual elements such as graphs, tables, or geometric figures.Table 2 reports subject distributions and averages for questions posed and answers expected per problem.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 Meta-Evaluation Framework (µ-MATH)

  • µ-MATH uses four model-generated solutions per text-only U-MATH problem and selects 271 problems, yielding 1,084 samples intended as a robust, challenging judge test rather than a representative U-MATH distribution.Correctness combines manual labeling with formal-equivalence checks; negative API responses, about 40% of cases, receive expert labeling because extraction and expression conversion can be imperfect. Models judge a solution against a problem and reference answer, with macro-averaged F1 as the primary metric.Source: §3.3 · pp. 4–5

4 Experiments and Results

Reasoning models outperform standard-inference models on U-MATH, but visual-task performance remains substantially below text-only performance. The experiments use controlled generation settings and an LLM judge for scoring.

4.1 Experimental Setup

  • The experiments use single-generation decoding, with 4,096-token limits for non-reasoning models and 32,768-token limits for reasoners; o-series models retain their default nonzero temperature.U-MATH solutions are evaluated with chain-of-thought prompting and o3-mini as judge.Source: §4.1 · p. 5

4.2 U-MATH Results

  • 86.8% U-MATH, 93.1% U-MATHT, and 58.5% U-MATHV are the top scores for reasoning models, versus 67.2%, 71.7%, and 47.0% for standard-inference models, respectively.The supplied results identify the three benchmark variants as overall, text, and visual performance.Source: §4.2 · p. 5
  • DeepSeek-V3 reaches 69.3% U-MATHT, close to Gemini 1.5 Pro at 71.7%, and DeepSeek-R1 reaches 91.3%, near o1 at 93.1%.For visual problems, the open-proprietary gap is more pronounced; the best open-weight results in the smaller and larger groups are Qwen2-VL 7B at 27.1% and Qwen2-VL 72B at 43.9%.Source: §4.2 · p. 5

Model U-MATH

U-MATH results show strong differences across model types and modalities, while µ-MATH evaluations find that judging solutions remains difficult and exhibits model-specific patterns.

Model U-MATH

  • Gemini leads the proprietary category across scales: Gemini 1.5 Flash scores 42.5%, Gemini 1.5 Pro 47.0%, and Gemini 2.0 Flash Thinking 58.5%.Source: Model U-MATH · p. 6
  • U-MATHV scores are consistently lower than U-MATHT, and adding visual input can also reduce textual performance across several models.Reported drops include Mistral and Pixtral Large from 48.1% to 42.9%, smaller Llama models from 26.1% to 18.6%, and QwQ and QVQ Preview from 71.8% to 59.3%.Source: Model U-MATH · p. 6
  • Math specialization can outweigh model size: Qwen2.5-Math 7B surpasses Llama-3.1 70B, while Qwen2.5-Math 72B performs on par with DeepSeek-V3.DeepSeek-V3 is a 685B mixture-of-experts model.Source: Model U-MATH · p. 6
  • Continuous finetuning raises U-MATH accuracy by 2.9% for Llama-3.1 70B to Nemotron 70B and 5.2% for Qwen2.5-72B to Athene-V2 72B.Source: Model U-MATH · p. 6

4.3 Meta-Evaluation (µ-MATH) Results

  • Manual CoT specifies steps to follow; it generally maintains or improves non-reasoner judgment, except for Llama models, while reasoners remain largely unaffected.Gemini 1.5 models gain more than 10% F1, whereas Llama models have more inconclusive judgments.Source: §4.3 · p. 7
  • Reasoning judges reach 90.1% F1, but even top systems remain fallible, and systematic judgment errors cannot be overcome simply with more data.Non-reasoner maximum F1 is 81.5%; judgment errors directly limit evaluation precision.Source: §4.3 · p. 7
  • Problem-solving strength does not ensure judgment strength, while o1 and o3-mini are among the best-performing and most balanced systems on TPR-TNR parity.The reported differences suggest a trade-off between problem-solving and judgment skills.Source: §4.3 · p. 7
  • Judges perform better on Llama solutions and worse on Qwen solutions, with the strongest author bias among smaller judges using AutoCoT.Bias is reduced with more capable judges and, for non-reasoners, with CoT; no noticeable self-judgment effects are observed.Source: §4.3 · p. 7

5 Conclusion

U-MATH and µ-MATH provide benchmarks for university-level mathematical problem-solving and solution judgment. Experiments expose weaknesses in visual reasoning and persistent difficulty and biases in LLM judgment.

5 Conclusion

  • U-MATH comprises 1,100 unpublished university-level problems from real teaching materials across six subjects, with 20% involving visual elements.µ-MATH is a U-MATH-derived dataset for assessing LLM judges.Source: §5 · p. 7
  • Maximum U-MATH accuracy is 93.1% on text-only tasks and 58.5% on visual tasks, with visual reasoning often degrading textual performance.The paper also reports that reasoning-first training, continuous finetuning, and mathematical specialization boost performance.Source: §5 · p. 7
  • Judgment differs from problem-solving and remains challenging: even the most capable models peak at 90.1% F1, with pronounced biases and instabilities also observed.Source: §5 · p. 7

Limitations

U-MATH does not span all advanced mathematical subjects, and its textual tasks approach saturation with reasoning systems. Visual tasks are not included in meta-evaluations, while accuracy and LLM-judge errors constrain evaluation detail and reliability.

Limitations

  • The benchmark omits some advanced mathematical subjects, and its textual tasks approach saturation with reasoning systems.The authors identify complex analysis as an example of an advanced topic for expansion.Source: Limitations · p. 9
  • Visual problems make up 20% of the benchmark, and visual problems are not covered by the meta-evaluations.The authors say the visual fraction limits the scope of visual reasoning evaluations.Source: Limitations · p. 9
  • Accuracy discards signal needed for finer-grained analysis, while LLM-judge errors and biases require further principled mitigation.The authors describe quantifying these errors as only a first step.Source: Limitations · p. 9

Future Work

  • Future work includes partial-credit assessment, controlling for auto-evaluation uncertainty and bias, and adversarial task creation or modification.The authors present adversarial approaches as a possible way to address saturation besides costly new-data curation.Source: Limitations · p. 9

Ethics Statement

The authors report collecting U-MATH and µ-MATH with appropriate permissions and excluding personal, proprietary, and sensitive content.

Ethics Statement

  • U-MATH and µ-MATH contain mathematical problems and solutions without sensitive content, and were collected with appropriate permissions.The authors state that no personal or proprietary information is included.Source: Ethics Statement · p. 9
  • The datasets and evaluation code are open-sourced under suitable licenses to support transparency and research advancement.Source: Ethics Statement · p. 9
  • The authors report no known conflicts of interest.Source: Ethics Statement · p. 9

Reproducibility Statement

The reproducibility statement says the datasets and evaluation code will be available on GitHub and points readers to descriptions of data processing, experimental setup, and prompts.

Reproducibility Statement

A.1 U-MATH Sample Problems

The appendices show examples of U-MATH and µ-MATH problems, summarize U-MATH topic distribution, and provide prompts and model-comparison figures and tables.

A.1 U-MATH Sample Problems

  • Figure 4 presents text-only and visual U-MATH problem examples with their subjects and golden answers.Source: Figure 4 · p. 16

A.2 U-MATH Sample Problem with Reference Solution

  • Figure 5 shows a U-MATH problem alongside its reference solution and golden answer.Source: Figure 5 · p. 17

A.3 µ-MATH Sample Problem

  • Figure 6 illustrates a µ-MATH problem by comparing the golden answer with an LLM-generated answer.Source: Figure 6 · p. 18

B U-MATH Topic Distribution

C.1 Prediction Prompt

  • Figure 7 shows the inference prompt used to sample solutions from problem statements.Source: Figure 7 · p. 20

C.2 Judgment Prompts

  • Figure 9 presents the default CoT judgment prompt used for U-MATH evaluations; Figure 8’s AutoCoT variant is only for µ-MATH experiments.Figure 10 shows the prompt for extracting the final verdict from the judge’s output.Source: Figure 9 · p. 21, Figure 8 · p. 21, Figure 10 · p. 22

D U-MATH Visual Comparison

  • Figure 11 compares selected top-performing models on U-MATH, U-MATHText, and U-MATHVisual.Source: Figure 11 · p. 23

E µ-MATH Inconclusive Judgment Rate

  • Table 7 reports inconclusive-judgment percentages by model and prompting scheme on µ-MATH.Source: Table 7 · p. 24

F Problem-solving vs. Judgment, Conservatism vs. Leniency, Reasoning vs. Coherence

Problem-solving strength does not necessarily yield better judgment: the models show a possible trade-off between these skills and between formal reasoning and general coherence. The section connects judgment styles to different true- and false-positive patterns and illustrates a correct judgment on a calculus example.

F Problem-solving vs. Judgment, Conservatism vs. Leniency, Reasoning vs. Coherence

  • Figure 12 compares textual problem-solving with judgment; stronger problem-solving does not necessarily mean stronger judgment, and non-reasoners show a possible inverse relationship beyond a threshold.The observed pattern is described as a potential trade-off, not a universal rule.Source: Figure 12 · p. 25, F Problem-solving vs. Judgment,… · p. 25
  • Claude’s strong judgment despite weaker problem-solving and Qwen-Math’s stronger problem-solving but weaker judgment are proposed as extremes of a reasoning–coherence trade-off.Qwen-Math struggles with instruction comprehension, formatting, and maintaining task structure; Claude handles these better, sometimes at the expense of in-depth reasoning.Source: F Problem-solving vs. Judgment,… · pp. 25–26
  • Lenient judges tend toward higher TPR and lower TNR, while conservative judges tend toward higher TNR and lower TPR; Claude and Qwen-Math are opposite non-reasoner examples with approximately equal F1-scores.The text associates leniency with following solutions and longer derivations, and conservatism with structured, label-anchored judgments.Source: F Problem-solving vs. Judgment,… · p. 26
  • GPT-4 and Gemini 1.5 models are described as conservative, Qwen models as lenient, and Qwen2.5-Math as more balanced than Qwen2.5; smaller models tend to show stronger bias.The authors suggest tendencies may reflect training data and note that reduced capability can worsen bias, with conservative models losing TPR and lenient models losing TNR.Source: F Problem-solving vs. Judgment,… · p. 26
  • Reasoner systems tend toward higher TPR, and o1 and o3-mini are among the most balanced models; R1 is the exception, with traces often driving conservative judgments.The authors suggest better-balanced training may be promising for reasoning systems; they also state that coherence and reliability are needed to apply problem-solving skills successfully.Source: F Problem-solving vs. Judgment,… · p. 26

G µ-MATH Judgment Examples

  • In the differential-calculus example, the displayed simplification yields equal expressions; the passage concludes they are identical and the solution is correct, consistent with the golden judge’s “Yes” verdict.Source: G µ-MATH Judgment Examples · pp. 27–28
Loading 2412.03205v4…