Source-linked AI summary
SKYLENAGE Technical Report: Mathematical Reasoning and Contest-Innovation Benchmarks for Multi-Level Math Evaluation
Hu Wei, Ze Xu, Boyu Yang, Linlin Miao, Weiqi Zhai, Yihan Li, Zixuan Li, Zhijun Wang, Boya Wang, Jianwei Yu, Jialing Yuan, Xiaoyue Zhang, Cheng He, Minglei Chen, Zifan Zhang, Qianhui Li, Wei Wang, Xiang Xu
TL;DR
Existing math benchmarks can saturate and obscure subject, grade-level, and hardest-slice differences among models. SKYLENAGE introduces complementary structure-aware diagnostic and multi-stage contest benchmarks and evaluates 15 LLM variants. Contest-style results separate leading systems, while the reasoning set reveals hardest-slice robustness gaps.
Problem
Existing math benchmarks can saturate and compress heterogeneous capabilities into aggregate scores, obscuring frontier differences and subject- or grade-level variation.
Source: §1 · p. 1Method
The paper introduces complementary structure-aware reasoning and multi-stage contest benchmarks and evaluates 15 LLM variants under a unified protocol.
Source: §1 · p. 2Results
SKYLENAGE-MATH separates leading systems, with its strongest model reaching 44.0%, while SKYLENAGE-REASONINGMATH shows leaders retain about 37% on the hardest quintile versus mid-tier systems at 15% or less.
Source: §6 · pp. 12–13Takeaways & Limitations
The two benchmarks provide complementary views of contest difficulty and structural robustness, with results showing that aggregate scores alone are insufficient to characterize mathematical reasoning.
Source: §1 · p. 2SKYLENAGE-MATH raw items are withheld, and the current analyses emphasize exact-match outcomes rather than step-level verification or partial credit.
Source: §7 · pp. 13–14
Abstract
from arXiv · showhide
Large language models (LLMs) now perform strongly on many public math suites, yet frontier separation within mathematics increasingly suffers from ceiling effects. We present two complementary benchmarks: SKYLENAGE-ReasoningMATH, a 100-item, structure-aware diagnostic set with per-item metadata on length, numeric density, and symbolic complexity; and SKYLENAGE-MATH, a 150-item contest-style suite spanning four stages from high school to doctoral under a seven-subject taxonomy. We evaluate fifteen contemporary LLM variants under a single setup and analyze subject x model and grade x model performance. On the contest suite, the strongest model reaches 44% while the runner-up reaches 37%; accuracy declines from high school to doctoral, and top systems exhibit a doctoral-to-high-school retention near 79%. On the reasoning set, the best model attains 81% overall, and hardest-slice results reveal clear robustness gaps between leaders and the mid-tier. In summary, we release SKYLENAGE-ReasoningMATH and report aggregate results for SKYLENAGE-MATH; together, SKYLENAGE provides a hard, reasoning-centered and broadly covering math benchmark with calibrated difficulty and rich metadata, serving as a reference benchmark for future evaluations of mathematical reasoning.
1 Introduction
Existing math benchmarks can saturate and hide differences in subject strengths, grade-level resilience, and hardest-slice performance. SKYLENAGE introduces complementary diagnostic and contest-style benchmarks and reports that model leadership and accuracy vary across subjects and difficulty levels.
Benchmarks
- SKYLENAGE-REASONINGMATH contains 100 structure-first problems with item metadata, while SKYLENAGE-MATH contains 150 contest-style problems across four stages and seven subjects.The diagnostic set includes length, numeric density, and symbolic complexity metadata; the contest set permits multi-label subject annotations.Source: §1 · p. 2
Evaluation
- Fifteen LLM variants are evaluated under a unified protocol, with subject- and grade-conditioned analyses.The setup combines chain-of-thought prompting, small-sample self-consistency, standardized answer extraction, and exact-match grading with numeric tolerance.Source: §1 · p. 2
Findings
- Models show stable leader–mid–tail tiers, fragmented subject leadership, and steep accuracy declines from high school to doctoral problems.These findings indicate that a single aggregate score does not characterize mathematical reasoning ability.Source: §1 · p. 2
Benchmark strategy
- Both tracks are positioned as living benchmarks, pairing a frozen static core for comparability with controlled dynamic variants for robustness stress tests.The paper also reports analyses of subject specialization, grade-band resilience, and structural difficulty.Source: §1 · p. 2
Release
- SKYLENAGE-REASONINGMATH is publicly released with metadata and graders, while SKYLENAGE-MATH is limited to aggregate analyses due to sensitivity.The authors describe this as balancing evaluation value with reproducible research.Source: §1 · p. 2
2 Related Work: Mathematical Evaluation of LLMs
Prior math resources range from elementary word problems to contest-style reasoning, while many evaluations provide limited fine-grained comparison. Cross-benchmark results show differing frontier separation and rotating champions across suites.
2 Related Work: Mathematical Evaluation of LLMs
- Math benchmarks span short arithmetic word problems and contest-style items that stress multi-step symbolic reasoning.GSM8K provides a grade-school baseline; MATH covers competition-level problems, while SVAMP introduces crafted variants to reduce reliance on superficial cues.Source: §2 · pp. 2–3
2.1 Cross-Benchmark Comparisons
- Cross-benchmark accuracy covers 14 models with public numbers; GPT-5-Chat-0807 is omitted because comparable public scores are unavailable.The reported macro mean averages available benchmarks for each model, ignoring missing entries.Source: §2.1 · p. 3, Table 2 · p. 4
- 82.0 is GPT-5-20250807’s macro mean, ahead of Grok-4-0709 at 79.6 and Gemini2.5-Pro-0617 at 77.3.The reported +2.4-point advantage over Grok-4-0709 persists despite saturation on some benchmark columns.Source: §2.1 · p. 3
- AIME-style contests remain sensitive to frontier differences, whereas MATH-500 strongly saturates; GPQA and MMLU-Pro capture breadth beyond mathematics.AIME25 spans 80.3 points (99.6 to 19.3), while MATH-500 spans 14.2 points (99.4 to 85.2) with a 2.7-point top–runner gap (99.4 vs. 96.7).Source: §2.1 · p. 3
- Champion leadership rotates across suites: GPT-5-20250807 leads four of six columns, while Grok-4-0709 leads GPQA.The heatmap places benchmarks in rows and models in columns, with stars marking per-benchmark champions; ties are allowed.Source: §2.1 · p. 4, Figure 1 · p. 4
3 Dataset Construction
SKYLENAGE-REASONINGMATH is designed around structure-first reasoning, contamination control, and checkable item features. SKYLENAGE-MATH adds expert-curated contest problems stratified by academic stage and subject.
3.1 SKYLENAGE-REASONINGMATH: Design Goals, Sources, and Anti-Contamination
- The reasoning set combines human authoring, rule-based generation, and structure-preserving rewrites, with normalization and bilingualization for consistent style and difficulty.The pipeline applies string-, semantic-, and template-level contamination checks, then validates items and pilots difficulty before freezing the set.Source: Figure 2 · p. 5, §3.1 · pp. 5–6
- SKYLENAGE-REASONINGMATH targets decomposable structure-first problems, prioritizing logic, constraints, number theory, combinatorics, and spatial reasoning over rote computation.Its design also prioritizes low-frequency patterns and verifiable intermediate assertions.Source: §3.1 · p. 5
- Each reasoning item carries subject and structural metadata, process hooks, and a pilot-calibrated difficulty score.Features include length, numeric-token density, symbolic-token count, constraint count, and branching factor; stored invariants support process checks alongside exact-match grading.Source: §3.1 · pp. 5–6
3.2 SKYLENAGE-MATH: Curation Protocol and Dataset Characteristics (150)
- SKYLENAGE-MATH comprises 150 expert-authored or selected contest problems stratified across HS, UG, GR, and PhD stages; raw items are not released.The authors cite protection of sensitive or licensed content and preservation of future evaluation value as reasons for publishing aggregate analyses instead.Source: §3.2 · p. 6
- Contest items use a seven-subject taxonomy and mix single-skill questions with cross-topic composites to reflect contest problems.Dataset-facing analyses include stage distributions, subject coverage, multi-label composites, answer types, and subject-by-stage performance.Source: §3.2 · p. 6
- The contest set targets multi-lemma reasoning, diagram-free geometry, constructive number theory, and discrete structures.Its four-stage stratification is intended to expose top-tier separations where prior public sets saturate; the taxonomy supports downstream routing and ensemble analysis.Source: §3.2 · p. 6
4 Evaluation Protocol
The evaluation uses standardized prompting, answer normalization, and exact-match grading across models. The reasoning-set results show that overall accuracy and hardest-quintile performance distinguish systems differently.
Protocol
- The protocol uses chain-of-thought prompting, standardized answer extraction, and binary exact-match grading after normalization.Floating-point results receive a 10^-6 tolerance unless an item specifies an exact form; multi-subject contest items receive full credit under each tagged subject.Source: §4 · p. 6
Results
- 81% is GPT-5-20250807’s overall accuracy on Reasoning-100, while GPT-5-Chat-0807 leads the hardest quintile at 35%.GPT-5-20250807 and Qwen3-235B-A22B-2507 each score 30% on Q5.Source: Figure 3 · p. 7
Extensions
- Future releases will add process-aware checks for step validity, constraint fidelity, and verifier agreement alongside exact-match scores.The checks are intended as complementary CoT-based scores based on minimal per-item hooks.Source: §4 · p. 7
5 Dataset Analysis
The analyses show distinct model tiers and subject strengths, alongside sharper separation as problem structure and contest difficulty increase. Hardest-slice accuracy, answer form, and grade scaling reveal where performance diverges beyond aggregate scores.
5.1.1 Overall results and hardest-slice accuracy
- 81% overall accuracy accompanies Q5 retention of ≈0.37 for the flagship and ≈0.38 for Qwen, versus ≈0.27 for Grok and ≤0.145 for the mid-tier.The hardest slice magnifies gaps between systems with similar top-line scores.Source: §5.1.1 · p. 7
5.1.2 Subject- and difficulty-wise profiling
- The flagship leads Grok in Combinatorics (92.9% vs. 71.4%), Probability (83.3% vs. 50.0%), and Number Theory (81.0% vs. 52.4%); Qwen leads Geometry (75.0% vs. 68.8%).At Q5, the flagship and Qwen retain 37–38% of baseline, versus Grok’s 20% and GPT-oss-120b’s ≤15%.Source: Figure 4 · p. 8
5.1.5 Hardest items (diagnostics)
- The hardest 10 items are dominated by Algebra (6/10) and Number Theory (3/10), with mean accuracy ≤11.8%.The items include short, digit-dense problems and long symbol-rich composites; their failure modes include arithmetic or normalization slips and step drift.Source: Table 3 · p. 9, §5.1.5 · pp. 9–10
5.1.3 Subject × model heatmap
- A per-subject oracle would harvest gains over any single model, largest in discrete-heavy cells where the leading margin exceeds 20% relative.Qwen leads the flagship in Geometry, while the flagship holds edges in Probability and Number Theory.Source: §5.1.3 · pp. 8–9
5.1.4 Structure vs. performance
- Numeric density separates model families more sharply than length or symbolic complexity, whose error associations are weak (r≈0.2).Errors rise ∼+90% for an open-weight 120B variant and ∼+30% for a strong proprietary baseline as digit density increases; the flagship rises ∼+18%, while Qwen is near-flat or slightly negative.Source: §5.1.4 · p. 9, Figure 6 · p. 9
5.2.1 Composition and Meta-Overview
- 44.0% for GPT-5-20250807 leads 37.3% for Grok-4-0709; SKYLENAGE-MATH accuracy falls from 26.3% at HS to 14.1% at PhD.The three-tier landscape persists across resamplings and decoding settings.Source: §5.2.1 · p. 10, Figure 7 · p. 10
- The flagship leads contest performance at 44.0%, ahead of Grok at 37.3% and Qwen at 31.3%; PhD accuracy is 14.1% versus 26.3% at HS.The leader–tail spread is 44.0% versus 10.7%, and multi-lemma planning and canonicalization pressures amplify separations.Source: §5.2.1 · p. 10, Figure 7 · p. 10
5.2.2 Subject accuracy and champions
- Subject leadership fragments: the flagship peaks in Combinatorics (58.3%) and Graph Theory (40.7%), Grok leads Geometry (44.9%), and Qwen is competitive in Probability (42.9%).The reported subject maxima motivate per-subject routing, with the largest marginal gains in high-grade discrete regions.Source: §5.2.2 · p. 10, Figure 8 · p. 11
6 Discussion
The discussion emphasizes that the benchmarks expose differences that aggregate accuracy can miss, including answer-process deficiencies, grade-related separation, and rotating subject strengths. It also reports strong alignment between contest performance and HLE.
Discussion
- Correct final answers can arise from shortcutting or invalid intermediate steps, especially on Q5 and structure-heavy items; future releases will add process-based signals alongside accuracy.Examples include unsupported trigonometric claims and infeasible grid-maze move sequences.Source: §6 · pp. 11–12
- Accuracy is lower for symbolic/derivational answer forms than for numeric answers.Figure 10 summarizes this answer-type comparison.Source: Figure 10 · p. 12
- 44.0% for the strongest model versus 37.3% for the runner-up yields stable top-end separation, while the three-tier pattern persists despite saturation on other public math suites.The leader–tail spread exceeds +300% relative.Source: §6 · p. 12
- Accuracy drops from 26.3% at HS to 14.1% at PhD (−46.4%); the top model retains roughly 0.79 of HS performance at PhD, versus about 0.50 for mid-tier systems.At GR/PhD, leader–mid separations commonly exceed 15 points.Source: §6 · pp. 12–13
- Subject leadership rotates: the flagship is strongest in discrete domains, Qwen leads in Geometry on SKYLENAGE-REASONINGMATH, and contest-suite strengths also differ across models.The reported rotations support subject-aware routing or small ensembles.Source: §6 · p. 13
- On SKYLENAGE-REASONINGMATH, leaders retain about 37% of baseline at Q5, versus ≤15% for mid-tier systems; numeric density separates families more than sheer length.The flagship shows modest sensitivity to digit density, while Qwen trends slightly more stable.Source: §6 · p. 13
- SKYLENAGE-MATH aligns strongly with HLE (Pearson r=0.9226, R2=0.851); each +1 HLE point predicts about +1.34 points on SKYLENAGE-MATH.The passage interprets this alignment as near-linear gains across models on extended derivations and contest-style problem solving.Source: §6 · p. 13, Figure 11 · p. 13
7 Limitations
The 150-item analyses have coverage and scoring limits, omit operational-cost measures, and currently emphasize final-answer accuracy. Future releases plan dynamic variants and process-based scoring.
7 Limitations
- Calculus and Probability are less represented than Geometry and Algebra, which can saturate champion cells; ties are marked and extreme cells warrant caution.Step-level verification and partial credit are out of scope.Source: §7 · p. 13
- Exact-match grading leaves step-level verification and partial credit out of scope, while current analyses emphasize final-answer outcomes.Forthcoming releases will incorporate process-based scoring to capture intermediate reasoning quality more fully.Source: §7 · pp. 13–14
- Latency, token cost, and verbosity are intentionally omitted from the 150-item analyses.Source: §7 · p. 13
- Public math sets may suffer training contamination; the authors mitigate this through curation and plan to expand with newly authored items.Source: §7 · p. 13
- Both tracks are planned to gain dynamic variants, and ReasoningMATH will add process-based scoring such as step validity and verifier agreement.Source: §7 · p. 14
9 Ethics Statement
The authors state that all problems are curated for research and offer a filtered release if licensed materials are requested for removal.
9 Ethics Statement
- All problems are curated for research; the authors will provide a filtered release if licensed materials are requested for removal.Source: §9 · p. 14
10 Conclusion
SKYLENAGE-ReasoningMATH and SKYLENAGE-MATH provide complementary high-difficulty evaluations, targeting structural reasoning and contest-style difficulty with grade scaling. The authors report shortcomings in intermediate reasoning despite strong answer accuracy and plan dynamic variants and process-based scoring.
10 Conclusion
- The two tracks complement one another: ReasoningMATH probes multi-constraint reasoning with item-level structural annotations, while MATH uses contest-style difficulty and explicit grade scaling.Source: §10 · p. 14
- Despite strong answer accuracy on SKYLENAGE-ReasoningMATH, the authors observe frequent shortcutting and chance correctness unsupported by valid inferential steps.Source: §10 · p. 14
- Both tracks are planned as dynamic benchmarks pairing a frozen static core for comparability with controlled variants for robustness stress testing.Source: §10 · p. 14
- ReasoningMATH will introduce process-based scoring, including step validity and verifier agreement, to distinguish correct-by-guess from correct-by-reasoning and provide step-level diagnostics.Source: §10 · p. 14
Supervision
The supplied passages identify prior work on mathematical word problems, rationale generation, and reasoning methods, but provide bibliography fragments rather than a substantive account of supervision.
Supervision
- Cobbe et al. (2021) present training verifiers to solve math word problems.Source: Supervision · p. 15
- Ling et al. (2017) describe program induction by rationale generation for algebraic word problems.Source: Supervision · p. 15
- Wang et al. (2022) report that self-consistency improves chain-of-thought reasoning in language models.Source: Supervision · p. 15
- Wei et al. (2022) describe chain-of-thought prompting as eliciting reasoning in large language models.Source: Supervision · p. 16
A.1 Supplement for Cross-Benchmark
Across public benchmarks, the flagship leads with consistent gaps, but no model dominates every axis; alignment with HLE varies across benchmarks. The case studies show how structure-first checks expose errors in constraints, indexing, and answer construction.
A.1.1 Macro structure and separation
- The flagship leads the macro mean with persistent gaps over the runner-up and mid-band, while AIME24/25 provide the broadest frontier spread and MATH-500 compresses top scores.GPQA and MMLU-Pro fall between those benchmarks in discriminative spread.Source: §A.1.1 · p. 17
A.1.2 Normalized profiles and complementarity
- Radar profiles show no model dominates all axes: many spike on MATH-500/AIME and dent on HLE, reflecting specialization rather than uniformly balanced performance.The profiles use row-wise min–max normalization.Source: §A.1.2 · p. 17, Figure 1 · p. 17
A.1.3 Practical implication
- Use AIME, GPQA, and HLE to discriminate frontier systems, treat MATH-500 as a reliability gate, and pair contest and knowledge/long-form specialists for mixed workloads.Source: §A.1.3 · p. 17
A.1.4 Benchmark-Benchmark Alignment with HLE
- r≈0.90 (GPQA) and 0.87 (AIME25) show strongest HLE ordering agreement; MATH-500 is weakest at r≈0.49, with its slope and intercept indicating ceiling compression.AIME25’s slope is 2.829; GPQA’s slope is 0.906 with an upward offset of +64.18.Source: §A.1.4 · p. 18
A.2.1 Case Study 1: A BFS Maze Item (Structure-First Grid Reasoning)
- The maze item exposes search-discipline failures: models may misread indexing, cross a wall, mismatch move tokens to a path, or return a valid but nonminimal route.The item diagnoses discrete-constraint handling and consistency between planning and move-string rendering.Source: §A.2.1 · p. 20
A.2.2 Case Study 2: A Structure-First Trigonometry Item
- The triangle item tests whether models preserve angle constraints, use a short nonnegativity bound, and provide a constructive witness; common errors include overcomplicated algebra and unsupported attainability claims.A symmetric perturbation around the equilateral configuration can demonstrate feasibility by construction.Source: §A.2.2 · pp. 20–21
- The roadmap extends structure-first diagnostics with parameterized witnesses, adversarial constraint variants, and process-checkable annotations to identify specific reasoning lapses.The suite targets low-computation problems that reveal tool choice and verifiable, constructive conclusions.Source: §A.2.2 · p. 21