Source-linked AI summary
AMO-Bench: Large Language Models Still Struggle in High School Math Competitions
Shengnan An, Xunliang Cai, Xuezhi Cao, Xiaoyu Li, Yehao Lin, Junlin Liu, Xinxuan Lv, Dan Ma, Xuanlin Wang, Ziwen Wang, Shuang Zhou
TL;DR
AMO-Bench targets the saturation, memorization, and manual-verification limitations of existing mathematical reasoning benchmarks. It introduces 50 original, IMO-level-or-harder problems with final-answer grading and evaluates 26 LLMs. The best model reaches only 52.4% accuracy, while test-time scaling indicates substantial room for improvement.
Problem
Existing mathematics benchmarks face performance saturation, potential data leakage from competition-derived problems, and inefficient manual verification for proof-based tasks.
Method
AMO-Bench provides 50 human-crafted, originality-screened problems cross-validated to meet at least IMO difficulty standards, using answer-type-specific automatic grading.
Results
52.4% accuracy is achieved by GPT-5-Thinking (High), while most evaluated models score below 40%; top-tier reasoning models exceed 70% pass@32.
Takeaways & Limitations
Current LLMs still struggle with AMO-Bench, while increasing test-time compute reveals substantial potential for improving mathematical reasoning.
Takeaways & Limitations
AMO-Bench requires each problem to meet or exceed IMO difficulty standards, defining a deliberately narrow high-difficulty evaluation scope.
Abstract
from arXiv · showhide
We present AMO-Bench, an Advanced Mathematical reasoning benchmark with Olympiad level or even higher difficulty, comprising 50 human-crafted problems. Existing benchmarks have widely leveraged high school math competitions for evaluating mathematical reasoning capabilities of large language models (LLMs). However, many existing math competitions are becoming less effective for assessing top-tier LLMs due to performance saturation (e.g., AIME24/25). To address this, AMO-Bench introduces more rigorous challenges by ensuring all 50 problems are (1) cross-validated by experts to meet at least the International Mathematical Olympiad (IMO) difficulty standards, and (2) entirely original problems to prevent potential performance leakages from data memorization. Moreover, each problem in AMO-Bench requires only a final answer rather than a proof, enabling automatic and robust grading for evaluation. Experimental results across 26 LLMs on AMO-Bench show that even the best-performing model achieves only 52.4% accuracy on AMO-Bench, with most LLMs scoring below 40%. Beyond these poor performances, our further analysis reveals a promising scaling trend with increasing test-time compute on AMO-Bench. These results highlight the significant room for improving the mathematical reasoning in current LLMs. We release AMO-Bench to facilitate further research into advancing the reasoning abilities of language models. https://amo-bench.github.io/
1 Introduction
AMO-Bench addresses saturation, memorization, and manual-scoring limitations in existing mathematical reasoning benchmarks with 50 original, IMO-level problems designed for automatic grading. Results across 26 models show that current LLMs still struggle, despite promising test-time scaling potential.
- Motivation: Existing mathematics benchmarks are becoming less effective because top-tier models approach saturation and competition-derived data raises memorization concerns.Proof-based difficult questions additionally require manual expert verification, hindering efficient and consistent automated evaluation.
- Benchmark design: AMO-Bench contains 50 newly crafted problems cross-validated to meet at least IMO difficulty standards and screened for similarity to existing resources.An LLM-based filtering stage further excludes problems insufficiently challenging for current reasoning models.
- Benchmark design: Each problem requires a final answer rather than a full proof, enabling automatic grading through parser-based or LLM-based methods selected by answer type.This design balances grading cost and generalizability.
- Benchmark resources: Human-authored reasoning paths accompany final answers, supporting solution transparency, prompt engineering, and error analysis.The benchmark data and evaluation code are publicly available.
- Empirical motivation: 52.4% accuracy is the best result among 26 evaluated models, achieved by GPT-5-Thinking (High), while most models score below 40%.AMO-Bench also induces much higher output-token consumption than AIME24/25, underscoring its difficulty.
2 AMO-Bench
AMO-Bench is a 50-problem benchmark built through expert-led reviews for quality, originality, and Olympiad-level difficulty. It combines answer-type-specific grading with statistics on problem categories and annotated-solution lengths.
- Problems are independently designed by mathematics experts, with detailed solutions supplied for subsequent quality review.
- Each candidate undergoes expert review for semantic unambiguity, logical correctness, and alignment with knowledge typically covered in IMO-level competitions.
- Originality review combines 10-gram comparisons against existing datasets, web searches, and expert checks for similarity to past competition problems.
- Problems must meet or exceed IMO difficulty standards, while at least two advanced reasoning models must fail to solve them consistently during preliminary evaluation.Consistency is assessed over three samples per model; three correct samples indicate consistent solving ability.
- 50 problems span five categories: 13 Functions & Sequences, 12 Combinatorics, 11 Algebraic Equations & Inequalities, 9 Number Theory, and 5 Geometry.
- Parser-based grading handles numerical, set, and variable-expression answers, while descriptive answers use five-sample majority voting with o4-mini; manual checks report 99.2% accuracy.The benchmark contains 39 parser-graded problems and 11 LLM-graded descriptive problems.
3 Experiments
AMO-Bench evaluates diverse leading LLMs on exceptionally difficult mathematical problems, showing low accuracy but promising gains from greater reasoning effort and test-time computation.
- Experimental setup: The evaluation samples each model 32 times and reports their average performance as AVG@32.
- Main results: GPT-5-Thinking (High) reaches 52.4% AVG@32, while most evaluated models score below 40% on AMO-Bench.
- Reasoning efficiency: Higher-performing models generally produce longer outputs, with first-tier models above 40% AVG@32 using more than 35K completion tokens.
- Test-time scaling: AMO-Bench performance grows near-linearly with the logarithm of average output length when reasoning effort is controlled for GPT-5, o4-mini, and o3-mini.
- Potential for improvement: Increasing pass@k yields rapid gains through k=8 and gradual improvement afterward, with top-tier reasoning models exceeding 70% at pass@32.
4 Related Work
Mathematical reasoning benchmarks have progressed from early datasets toward Olympiad-level and research-oriented problems as performance on earlier tests saturates.
- Model performance on early benchmarks such as GSM8K and MATH has quickly reached saturation, motivating more challenging evaluations.
- One research direction evaluates Mathematical Olympiad-level problems requiring specialized knowledge and complex, intuitive reasoning.
- Another direction evaluates graduate-level examinations and advanced mathematical research through computationally intensive, abstract, or real-world tasks.
5 Conclusion
AMO-Bench is a 50-problem benchmark targeting Olympiad-level mathematical reasoning with original, automatically gradable questions, and current LLMs still struggle on it.
- AMO-Bench contains 50 human-crafted problems at mathematical Olympiad level or higher, all original and requiring only final answers for automatic, robust grading.
- Experimental results show that contemporary LLMs still struggle with the significant challenges presented by AMO-Bench.
- Further analysis identifies substantial opportunities for advancing current LLMs’ mathematical reasoning capabilities.
A Prompt Templates
The prompt templates standardize model answers and grading so outputs can be parsed and checked for mathematical equivalence.
- The query prompt requires a fixed answer prefix, a boxed LaTeX answer, and sufficient precision for decimal approximations.
- Decimal answers must be accurate to at least four decimal places in both the query and grading instructions.
- The grading prompt asks an LLM to classify whether a student answer is equivalent to the reference answer as Correct or Incorrect.
B Analysis of AVG@k
As sampling times increase, model performance becomes more stable, with reverse-order phenomena rarely appearing at 32 samples.
- At 32 sampling times, reverse-order phenomena rarely appear, indicating more stable average performance across models.Performance fluctuates less as the number of samples grows.
C Performance on AMO-Bench-P Subset
AMO-Bench-P contains 39 parser-based grading problems and generally yields slightly higher performance than the full benchmark because complex descriptive answers are harder to grade.
- AMO-Bench-P includes 39 parser-based grading problems, while Table 1 compares its AVG@32 performance with the full AMO-Bench.
- Performance on AMO-Bench-P generally exceeds performance on the full AMO-Bench.The subset excludes problems requiring complex descriptive answers.
- Problems requiring complex descriptive answers are inherently more challenging than problems with simple-format answers.