PaperlayerView exact revision on arXiv

Source-linked AI summary

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Guijin Son, Seungone Kim, Catherine Arnett, Hyunwoo Ko, Hyein Lee, Hyeonah Kang, Jiang Longxi, Jin Yun, JungYup Lee, Kyungmin Lee, Sam Yoosuk Kim, Sang Park, Seunghyeok Hong, SeungJae Lee, Seungyeop Yi, Shinae Shin, SunHye Bok, Sunyoung Shin, Yonghoon Ji, Youngtaek Kim, Hanearl Jung, Akari Asai, Graham Neubig, Sean Welleck, Youngjae Yu, Akshelin R, Alexander B. Ivanov, Boboev Muhammadjon, Chae Young Han, Christian Stump, Cooper R. Anderson, Dmitrii Karp, Dohyun Kwon, Dongryung Yi, DoYong Kwon, Duk-Soon Oh, Eunho Choi, Giovanni Resta, Greta Panova, Huiyun Noh, Hyungryul Baik, Hyungsun Bae, Inomov Mashrafdzhon, Jeewon Kim, Jeong-Rae Kim, Ji Eun Lee, Jiaqi Liu, Jieui Kang, Jimin Kim, Jon-Lark Kim, Joonyeong Won, Junseo Yoon, Junwoo Jo, Kibeom Kim, Kiwoon Kwon, Mario Kummer, Max Mercer, Min Hoon Kim, Minjun Kim, Nahyun Lee, Ng Ze-An, Nicolas Libedinsky, Rafał Marcin Łochowski, Raphaël Lachièze-Rey, Robert Auffarth, Ruichen Zhang, Sejin Park, Seonguk Seo, Shin Jaehoon, Sunatullo, Taewoong Eom, Yeachan Park, Yongseok Jang, Youchan Oh, Zhaoyang Wang, Zoltán Kovács

arXiv:2605.09063v3cs.CL

TL;DR

Research-level math benchmarks remain scarce and often narrow, limiting evaluation of advanced mathematical reasoning. SOOHAK addresses this gap with a 439-item benchmark authored by mathematicians, including Challenge and Refusal subsets. Models perform poorly on both research-level problem solving and recognizing ill-posed prompts, leaving substantial headroom.

  • Problem

    Research-level math benchmarks remain scarce and narrow in field or size, limiting broad comparison of mathematical capabilities across difficulty levels.

  • Method

    SOOHAK assembles Challenge and Refusal subsets through expert contributor recruitment, multi-stage collection and filtering, interviews, and separate refusal-split construction.

  • Results

    Performance falls steeply from SOOHAK-Mini to Challenge, while Refusal remains difficult; Gemini-3-Pro leads Challenge with 30.39% Avg@3 and GLM-5 leads Refusal with 49.49%.

  • Takeaways & Limitations

    SOOHAK provides a challenging evaluation of graduate-level and research-adjacent reasoning while exposing refusal on ill-posed prompts as a complementary capability gap.

  • Takeaways & Limitations

    SOOHAK was assembled under a roughly USD 550,000 budget and a four-month timeline, with noisy and incomplete difficulty labels and limited subfield coverage.

Abstract

from arXiv · show

Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM reasoning. Whereas olympiad-style problems measure step-by-step reasoning alone, research-level problems use such reasoning to advance the frontier of mathematical knowledge itself, emerging as a compelling alternative. Yet research-level math benchmarks remain scarce because such problems are difficult to source (e.g., Riemann Bench and FrontierMath-Tier 4 contain 25 and 50 problems, respectively). To support reliable evaluation of next-generation frontier models, we introduce Soohak, a 439-problem benchmark newly authored from scratch by 64 mathematicians. Soohak comprises two subsets. On the Challenge subset, frontier models including Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach 30.4%, 26.4%, and 10.4% respectively, leaving substantial headroom, while leading open-weight models such as Qwen3-235B, GPT-OSS-120B, and Kimi-2.5 remain below 15%. Notably, beyond standard problem solving, Soohak introduces a refusal subset that probes a capability intrinsic to research mathematics: recognizing ill-posed problems and pausing rather than producing confident but unjustified answers. On this subset, no model exceeds 50%, identifying refusal as a new optimization target that current models do not directly address. To prevent contamination, the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.

Organizing Team

The organizing team is presented as a roster of named contributors spanning two consecutive lists. The lists include Guijin Son, Seungone Kim, Akari Asai, Graham Neubig, Sean Welleck, and many others.

  • The first roster lists Guijin Son, Seungone Kim, Catherine Arnett, Hyunwoo Ko, Hyein Lee, Hyeonah Kang, and numerous additional contributors.
  • The first roster also includes Akari Asai, Graham Neubig, Sean Welleck, and Youngjae Yu among its named contributors.
  • The second roster continues with Akshelin R, Alexander B. Ivanov, Boboev Muhammadjon, Chae Young Han, Christian Stump, and many others.

1 Introduction

SOOHAK is a mathematician-authored benchmark designed to address contamination and saturation in existing mathematical reasoning evaluations. It combines a 340-item Challenge subset with a 99-item Refusal subset and shows substantial headroom for frontier models while remaining tractable for strong human solvers.

  • Motivation: Publicly sourced problems scale quickly but increase training-data overlap and accelerate benchmark saturation as frontier models improve.Fresh human-authored problems avoid contamination, but prior efforts were usually limited to one mathematical area or small datasets.
  • Benchmark design: SOOHAK contains 340 Challenge items and 99 Refusal items, with Challenge problems authored by 68 contributors, including 38 faculty members.The contributors also include 25 PhD students or postdoctoral researchers and 5 master’s or undergraduate IMO medalists.
  • Model results: 30.39%, 26.37%, and 10.39% are the Avg@3 scores of Gemini-3-Pro, GPT-5, and Claude-Opus-4.5, respectively, on SOOHAK Challenge.These results come from evaluations across eleven closed and open-weight systems.
  • Human baseline: 50.6% of 79 prompts were covered by aggregated human teams, indicating that the benchmark is challenging but tractable for strong human solvers.The 25 participants represented five teams and included IMO honorable mention recipients through gold medalists, mathematically trained undergraduates, and PhD-level researchers.

2 Related Work

Earlier math benchmarks initially measured LLM reasoning but have become less discriminative as models improve. Related work distinguishes olympiad-style benchmarks from research-level benchmarks and highlights contamination as a threat to reliable evaluation.

  • Earlier math benchmarks: <10% accuracy marked the poor performance of early models on MATH and GSM8K, but improved capabilities have made these benchmarks less discriminative at the frontier.This motivated newer benchmarks designed to track rapid progress and sometimes resist fast saturation.
  • Benchmark categories: Olympiad-style benchmarks emphasize multi-step solving in knowledge-contained settings, whereas research-level benchmarks probe advanced mathematical knowledge.Olympiad-style tasks typically avoid specialized background, admit short machine-checkable answers, and often derive from competition materials or unified suites.
  • Contamination: Publicly sourced exams, competitions, and published materials create contamination risks, and documented overlap can cause benchmark scores to overestimate true generalization.These contamination concerns arise because many benchmarks are assembled from sources that may have entered training data.

3 Data Collection

SOOHAK was assembled through a contributor-driven, multi-stage process combining originality requirements, automated and human quality control, model-gated routing, and final inclusion review. The collection also used contributor interviews to characterize Challenge construction and created a Refusal split from ill-posed submissions.

  • Contributor pool: 105 contributors provided accepted questions: 86 through the primary system across 31 organizations and 19 through the ScienceBench contribution group.Including ScienceBench contributors, the pool comprised 48% faculty, 23% graduate students or postdoctoral researchers, 25% undergraduates, and 5% undisclosed affiliation.
  • Collection pipeline: The five-stage pipeline comprised submission, automated screening, manual review, contributor-controlled opt-in, and final inclusion.Primary-system contributors affirmed original authorship, no AI use, and a copyright grant; LLM quality controls were audited by two human reviewers.
  • Collection pipeline: Submissions were attempted by baseline LLMs and routed through three model-gated collection gates requiring failure of progressively larger open models.The gates included Qwen3-8B and OpenThinker3-7B, then gpt-oss-20B and Qwen3-32B, followed by all large open models in the panel.
  • Contributor interviews: Interviews found that SOOHAK-Mini items could be written much faster, whereas a single SOOHAK Challenge problem often required one or more days of work.Challenge contributors commonly described submitting research-adjacent questions relying on folklore-level reasoning.
  • Refusal split: SOOHAK Refusal draws from quality-control rejections involving contradictions, missing assumptions, or no unique answer.A refusal item is correct only when a model diagnoses the flaw rather than confidently producing a numeric answer.

4 Language Model Evaluation Details

The evaluation covers eleven closed and open-weight language models with reasoning enabled, using three sampled responses per model–question pair. Performance is summarized by avg@3 and pass@3, with GPT-5-Mini judging parsed answers for mathematical equivalence.

  • Models: Eleven models are evaluated, spanning seven closed systems and four open-weight systems, with reasoning enabled for every model.The closed systems include Gemini-3-Pro, Gemini-3-Flash, GPT-5 Medium, GPT-5-Mini Medium, Claude-Opus-4.5, Claude-Sonnet-4.5, and Grok-4.1-Fast; open-weight systems include Qwen3-235B-A22B-thinking-2507, GPT-OSS-120B, Kimi-2.5, and GLM-5.
  • Implementation: GPT-5 performs best among GPT-5.1, GPT-5.2, and GPT-5 under identical configurations, so its results are reported.The comparison uses identical configurations for the three GPT-5 variants.
  • Sampling and metrics: For each model–question pair, three independent responses are sampled and evaluated using avg@3 and pass@3.Correctness is represented by binary indicators c_i,j across the three samples for each question.
  • Answer parsing and judging: Final answers are parsed and judged by GPT-5-Mini against gold answers using mathematical equivalence, without question text or solutions.The judge receives only the gold answer and parsed answer, then outputs a binary correctness label.

5 Results

SOOHAK performance drops sharply from Mini to Challenge, where closed models lead open-weight systems and many items remain unsolved. Results also show rotating subfield leaders and compute scaling that benefits Challenge more than Refusal.

  • Overall Performance: 30.39: Gemini-3-Pro leads Challenge Avg@3, while GPT-5 leads SOOHAK-Mini at 72.22 and GLM-5 leads Refusal at 49.49.The benchmark leaves 124 Challenge items unsolved by every evaluated model and 170 items unsolved or missed overall.
  • Overall Performance: 13.87: Kimi-2.5 posts the strongest open-weight Challenge score, trailing Gemini-3-Pro at 30.39 and GPT-5 at 26.37.On SOOHAK-Mini, Kimi-2.5 reaches 66.07 and GLM-5 63.11, compared with GPT-5 at 72.22 and Gemini-3-Pro at 71.70.
  • MSC subfield performance: Subfield leaders rotate by mathematical flavor: Gemini-3-Pro tops algebra, number theory, and analysis; Grok-4.1-Fast tops geometry and stochasticity.GPT-OSS-120B with hard reasoning and 81920 context wins MSC 15, the only subfield led by an open-weight model.
  • Compute scaling: 2.94 to 15.29: Qwen3 Challenge performance rises across model sizes from 0.6B to 32B, adding roughly three points per checkpoint after the first jump.Challenge also follows a comparable trajectory as test-time token budgets increase from the default 16,384 to 81,920 tokens, a 5× extension.
  • Compute scaling: Challenge scales roughly linearly with train- and test-time compute, whereas Refusal does not.The comparison is reported across the Qwen3 family and test-time token-budget settings.

6 Human Baselines

Human and LLM performance was compared on a shared 79-prompt evaluation set under distinct time- and token-budget conditions. Collectively, human teams covered 50.6% of questions, while Gemini-3-Pro was the only model to exceed that coverage at 60.8%.

  • Evaluation Design: 79 prompts comprised the shared evaluation set, including 49 Calibration and 30 Challenge problems, with harder questions intentionally upsampled.Challenge items came from narrower subfields and showed higher variance among strong solvers.
  • Evaluation Conditions: 4.5 hours was the nominal human time budget, with non-AI tools allowed but LLM use prohibited.Sessions were not fully standardized across participants.
  • Scoring: Pass@1 outcome-based scoring counted only correct final answers and awarded no partial credit to humans or LLMs.Both groups were evaluated on the same sampled prompts rather than the full dataset.
  • Results: 60.8% was Gemini-3-Pro’s accuracy, making it the only model to exceed combined human coverage at 50.6%.No single human profile fully covered the benchmark’s breadth, while five groups collectively covered all benchmark splits well.
  • Human Performance: Math Major with IMO experience and Math Major with IMO Gold achieved the highest human-participant scores, indicating contest training drove performance more than research experience.Math Researchers performed below these olympiad-trained undergraduate groups despite having deeper mathematical expertise.
  • Human Performance: Computational-tool use qualitatively correlated with better undergraduate-team outcomes, while shared-room engagement and collaboration strategy affected throughput and accuracy.These effects were observed alongside differences in skill and organizational behavior.

7 Discussion and Conclusion · A Author affiliations

SOOHAK evaluates graduate-level and research-adjacent mathematical reasoning through Challenge and Refusal subsets, while exposing persistent difficulty for frontier and open-weight systems. Its limitations reflect constrained development timelines and noisy difficulty labels, and its contributor superscripts map to listed institutions.

  • 7 Discussion and Conclusion: SOOHAK contains 340 Challenge items and 99 Refusal items for evaluating graduate-level and research-adjacent mathematical reasoning.The benchmark’s Challenge subset was newly authored by expert contributors.
  • 7 Discussion and Conclusion: 30.39% Avg@3 was the best model’s result on the Challenge subset.The Challenge subset remains difficult for frontier systems.
  • 7 Discussion and Conclusion: 13.87% Avg@3 was the strongest open-weight model’s Challenge result.The passage reports a sharper drop for open-weight systems.
  • 7 Discussion and Conclusion: SOOHAK and SOOHAK-Mini were assembled with roughly USD 550,000 or 800M KRW but only four months for recruitment, collection, review, and human baseline studies.The development process operated under unusual budget and timeline constraints.
  • 7 Discussion and Conclusion: Difficulty labels are noisy and incomplete proxies for benchmark value.The passage identifies this as the main lesson from the benchmark’s limitations.
  • 7 Discussion and Conclusion: Future efforts need early review infrastructure with explicit rubrics and incentive schemes rewarding more than raw difficulty.The passage also calls for globally scoped recruitment to broaden subf…
  • A Author affiliations: Numbered superscripts in the Organizing Team and Dataset Contributors blocks correspond to the institutions listed below.The affiliations section explains how the front-page superscripts should be interpreted.
  • A Author affiliations: The listed affiliations include OneLineAI, EleutherAI, Carnegie Mellon University, Dnotitia Inc., Saltlux Innovation, and SYSTRAN Korea.The passage continues with additional institutions, including KAIST and USC.

B Data collection details … D.1 Generation configuration

The appendix details Soohak’s funded collection pipeline, including contributor screening, split assignment, quality control, bilingual translation, and refusal-question construction. It also documents dataset composition, example-problem construction, evaluation extensions, and provider-recommended decoding configurations.

  • B.1 Funding context and Sovereign-AI background: The collection supported a South Korean MSIT Sovereign AI Foundation Model competition and ran from August to December 2025 under approximately $550,000 USD or KRW 800 million in government funding.The benchmark was designed for fair, contamination-resistant, and stable evaluation across multiple rounds, with confidentiality requirements and a late-2026 release.
  • B.2 Contributor terms, compensation eligibility, and submission statistics: 101 mathematicians and students submitted through the primary system, 86 had at least one accepted question, and the top five contributors submitted 434 questions.Compensation required successful vetting, confirmed split assignment, confidentiality agreements, and IP transfer to the project.
  • B.3 Item filtering: split assignment, manual reviewing, and quality control: Questions were routed through three model-gated collection gates, manually audited against reference solutions, and subjected to automated consistency checks with author re-solving when answers disagreed.The process corrected 87 submissions and banned multiple contributors for attempting to submit LLM-generated questions.
  • B.4 Bulk purchases and ScienceBench contribution protocol: ScienceBench contributed 112 additional Challenge problems from 19 contributors, bringing the full collection to 105 contributors.The contributors included 10 professors, 7 postdoctoral researchers, and 2 PhD students.
  • B.5 SOOHAK-Mini and Challenge creation strategies: SOOHAK-Mini contributors commonly remixed contest problems by preserving core invariants or tricks while changing parameters, constraints, or settings, whereas Challenge creation targeted harder material.The passage notes that stronger upper-math models could often solve such remixes quickly.
  • B.6 Problem-type composition: The reporting split contains 702 SOOHAK-Mini items, 340 Challenge items, and 99 Refusal items; SOOHAK-Mini includes 401 olympiad, 218 undergrad, 63 graduate, and 18 beyond items.Challenge includes 77 olympiad and 69 undergrad items, with the remaining items classified as graduate or beyond.
  • B.7 Translation pipeline: Approximately 92% of items were originally authored in English, and every item was translated into the other language through protected-LaTeX machine translation, professional post-editing, and independent review.The workflow enforced semantic faithfulness, terminology consistency, standardized notation, and minimal paraphrasing.
  • B.8 Refusal question sourcing and grading; D Evaluation extensions; D.1 Generation configuration: Refusal questions test whether systems recognize ill-posed or unanswerable prompts rather than produce persuasive but incorrect solutions, while evaluation extensions examine decoding, scaling, carefulness-adjusted ranking, and MSC-subfield performance.Main-results decoding follows provider recommendations: Gemini, GPT, and GPT-OSS use temperature 1.0, while remaining systems use temperature 0.6 where thinking variants are used.

D.2 Qwen3 size scaling … D.5 MSC subfield breakdown

Across Qwen3 models, benchmark performance generally improves with scale but can regress at the largest checkpoint, while extended context and reasoning effort especially improve Challenge. Carefulness-adjusted rankings penalize weak refusal performance, and MSC analysis reveals substantial variation across mathematical subfields.

  • D.2 Qwen3 size scaling: 35.75 to 70.80: SOOHAK-Mini Pass@3 scales monotonically across standard Qwen3 models from 0.6B to 32B parameters.Challenge Pass@3 also rises from 2.94 to 15.29 across the same sequence.
  • D.2 Qwen3 size scaling: 67.66 versus 71.23: Qwen3-235B-A22B-thinking-2507 regresses on SOOHAK-Mini relative to the 30B-A3B-thinking-2507 model.An 81920-token-context variant partially recovers to 70.23, suggesting output-length bottlenecking.
  • D.3 Test-time scaling with extended context and reasoning effort: 1.57 and 7.94 points: extending GPT-OSS-120B from medium to hard reasoning improves SOOHAK-Mini and Challenge, respectively.Further extending the context budget to 81920 tokens adds 4.13 points on SOOHAK-Mini and 3.24 points on Challenge.
  • D.3 Test-time scaling with extended context and reasoning effort: 15.00 to 22.3: Qwen3-235B-A22B-thinking-2507 shows the largest Challenge swing from extended context.The reported Pass@3 increase demonstrates that extended context can substantially improve Challenge performance for this model.
  • D.4 Carefulness-adjusted ranking: Three composite scores address how Refusal performance interacts with reasoning performance, because strong reasoning with weak refusal indicates confident errors on ill-posed prompts.The composites are defined from per-split Pass@3 values.
  • D.4 Carefulness-adjusted ranking: Avg-R incorporates Refusal as an equally weighted third dimension, reducing scores when refusal performance is low despite strong reasoning.SOOHAK-R pairs the hardest reasoning split with Refusal, while Capability is the unweighted mean of the two reasoning splits.
  • D.4 Carefulness-adjusted ranking: 3 ranks: GLM-5 rises from Capability to Avg-R, whereas Kimi-2.5 drops 3 ranks; GPT-5 takes the top Avg-R rank from Gemini-3-Pro.The ranking changes reflect differences in refusal care relative to reasoning capability.
  • D.5 MSC subfield breakdown: 69.8% mean and 80.6% best: MSC 40 is the strongest subject area, while MSC 16 is weakest at 14.6% mean and 48.6% best.MSC 26 and MSC 11 are also strong, whereas MSC 52 is weak, with 59.4% and 57.8% means respectively, and 75.1% and 71.9% best scores.

E Human team profiles

The study recruited five human teams with profiles spanning olympiad-trained computer scientists, mathematics students, elite olympiad competitors, informatics-oriented contestants, and mathematics and computer-science researchers. Teams differed in credentials, collaboration strategies, and, for two teams, attempted and correctly solved problem counts.

  • Team overview: Five teams (A–E) were recruited, with profiles summarized in Table 3 and per-team performance shown in Figure 3.The descriptions include credential breakdowns and, where available, collaboration strategies and attempt-vs-solve counts.
  • Team A: Team A comprised master’s and undergraduate computer-science students with extensive high-school olympiad experience, including 7 IMO Honorable Mentions and 2 IMO Bronze medals.The team also included 1 EGMO Bronze medal and 1 APMO Bronze medal.
  • Team B: Team B consisted of mathematics undergraduates with contest backgrounds, used division of labor with cross-checking, attempted 38 problems, and solved 30 correctly.Credentials included 1 WMTC Gold, 1 APMO Bronze, 1 IMO Silver, 1 KMS Silver, and 1 KMS Gold.
  • Team C: Team C combined elite olympiad and competitive-programming experience, emphasized parallelism and internal competition, attempted 44 problems, and solved 25 correctly.Its credentials included 2 IMO Gold, 1 IMO Silver, 1 APMO Bronze, 1 APMO Honorable Mention, 2 KMS Gold, 2 ICPC Seoul Gold, and 1 ICPC Bronze.
  • Team D: Team D was more informatics- and programming-oriented while retaining a contest-mathematics background, including SIMC 2.0 Champion and 2 KOI Bronze medals.Additional credentials included 1 KMS Silver, 1 KOI Silver, 1 ICPC Asia Pacific Bronze, and 1 ICPC Seoul Silver.
  • Team E: Team E comprised five PhD holders in mathematics and computer science, most of whom studied mathematics as undergraduates and published mathematical research.The team profile therefore combined advanced academic training with mathematical research experience.

F Discussion and limitations full retrospective

The retrospective identifies compressed timelines, limited review capacity, geographically constrained recruitment, difficulty-based incentives, and unique-integer grading as key constraints. It recommends earlier expert review, broader quality-aligned incentives, global recruitment, and richer evaluation formats.

  • Retrospective constraints: USD 550,000 (800M KRW) and a four-month end-to-end schedule constrained recruitment, review infrastructure, subfield coverage, and iterative piloting.The schedule covered proposal drafting, administration, contracting, recruitment, collection, review, and human baseline studies.
  • Retrospective constraints: Difficulty-based incentives increased measured difficulty but did not reliably capture question quality or benchmark value.Moderately difficult problems may track progress across model generations, whereas some very difficult problems are difficult for the wrong reasons.
  • Retrospective constraints: Limited rater hours delayed broad expert review, weakening early defenses against accessibility, notation, ambiguity, and evaluation-fragility problems.Experts were initially concentrated on question generation, with broader review capacity assembled only later.
  • Retrospective constraints: Domestic-first recruitment narrowed mathematical subfield and stylistic coverage, making global-scale recruitment effectively necessary for diverse advanced-mathematics benchmarks.The limitation persisted despite the quality of local experts.
  • Evaluation limitations: Unique-integer grading enables automatic scoring and reduces random guessing but restricts feasible problems and favors subfields with clean numeric answers.The format therefore limits the breadth of mathematical competence that can be measured.
  • Recommendations and broader impacts: The authors prioritize early rubric-based review, incentives aligned with multiple quality notions, and richer grading through formal, symbolic, or selective expert evaluation.They also warn that benchmark overinterpretation and incentives toward uniquely gradable problems are negative risks.
Loading 2605.09063v3…