Source-linked AI summary
From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities
Jiayi Kuang, Yinghui Li, Yunze Song, Keyu Chen, Zhifeng Shen, Yangning Li, Yidong Wang, Di Yin, Ruizhi Qiao, Xing Sun, Kai Jin, Ying Shen, Liang Lin, Philip S. Yu
TL;DR
Existing math benchmarks mainly assess final answers, leaving limited evidence about process-level failures and inherent agentic mathematical capabilities. The paper introduces AgenticMathBench, which aligns reusable mathematical atomic capabilities with planning, action, and feedback tasks across textual and multimodal settings. Experiments show that similarly accurate models can have substantially different agentic capability profiles, supporting process-level evaluation.
Problem
Most existing math benchmarks emphasize final answers, providing limited diagnostic evidence about process-level failures, logical rigor, and agentic capability gaps.
Method
AgenticMathBench aligns a taxonomy of mathematical atomic capabilities with Planning, Action, and Feedback and evaluates them through process-level tasks in text-only and multimodal contexts.
Results
Models with similar end-to-end performance exhibit substantially different agentic capability profiles, while planning and feedback evaluations expose process-level weaknesses.
Takeaways & Limitations
Process-level evaluation is needed to interpret LLMs’ agentic mathematical capabilities and guide future agent development.
Takeaways & Limitations
The taxonomy and task instantiations may not cover all mathematical agentic reasoning, especially long-horizon memory and iterative learning, and the multimodal subset is smaller than the text-only subset.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence. However, most existing math benchmarks evaluate only final answers. This outcome-oriented evaluation provides limited diagnostic value for identifying process-level failures or rigorous logic, failing to guide the transformation of LLMs into robust agents. To bridge this gap, we present a process-level benchmark designed to evaluate the inherent agentic mathematical reasoning abilities of LLMs. Our framework aligns problem-solving agentic behaviors with a structured taxonomy of reusable mathematical atomic capabilities. We design a comprehensive suite of planning, action, and feedback tasks across both textual and multimodal contexts, supported by an automated pipeline that synthesizes high-quality trajectories and produces fine-grained annotations via controlled LLM rewriting. Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles. This demonstrates that process-level evaluation is crucial for interpreting the true potential of LLMs and guiding the development of next-generation mathematical agents.
1 Introduction
Existing math benchmarks emphasize final correctness, offering limited diagnosis of reasoning processes and agentic capability gaps. AgenticMathBench addresses this by aligning mathematical atomic capabilities with planning, action, and feedback evaluation, revealing different agentic profiles among similarly accurate models.
- Most existing benchmarks emphasize final correctness, making problem-solving errors and overall logical validity difficult to diagnose.
- Existing evaluations also provide narrow or unsystematic coverage of mathematical capabilities and rarely align them with agentic behaviors.
- AgenticMathBench introduces a structured taxonomy of reusable mathematical atomic capabilities aligned with Planning, Action, and Feedback.
- The benchmark evaluates process-level tasks across multimodal and text-only settings using automated data collection, trajectory synthesis, filtering, and fine-grained annotation.
- Experiments show that models with similar end-to-end performance can exhibit substantially different agentic capability profiles.
2 Method
AgenticMathBench organizes mathematical atomic capabilities around Planning, Action, and Feedback, then builds process-level tasks and evaluation data to assess them separately.
- Benchmark framework: AMB aligns mathematical atomic capabilities with three agentic functions: Planning, Action, and Feedback.The framework targets intrinsic agentic intelligence rather than a fully deployed autonomous agent.
- Atomic capability system: The atomic capability system combines literature-derived definitions, dataset comparisons, expert consultation, and selective merging or removal of capabilities.The resulting taxonomy includes foundational, advanced reasoning, and mathematical meta-cognitive levels.
- Action: Action uses decoupled single-capability tasks to assess eight mathematical capabilities in isolation.Evaluation methods vary by target, including exact or structure-aware accuracy, set-based scores, Lean compilation, and semantic or judged measures.
- Data construction: The benchmark combines 27 filtered datasets with synthesized plan–action–feedback trajectories and retains 17.3% of trajectory data after filtering.Collected datasets include single-atom and composite-atom mathematical resources converted to unified formats and annotated across atomic abilities.
- Planning: Planning evaluates capability selection, ordered solution decomposition, and dynamic next-step generation while excluding execution correctness.Plans pair atomic capabilities with concrete sub-goals and must satisfy logical dependency.
- Feedback: Feedback evaluates correctness judgment, error localization, and repair over executed trajectories.Fix suggestions are assessed for consistency with the modified reasoning, next-step capability, and sub-task.
3 Experiment
Experiments show distinct process-level strengths and weaknesses across model families: commercial models lead planning and balanced action performance, while feedback and dynamic execution remain difficult.
- Planning results: Commercial models consistently outperform open-source models in capability planning, with Claude-4.5 achieving the highest F1 scores.DeepSeek-V3.2 and other strong open-source models can match or exceed some closed-source models on full solution planning.
- Planning results: Nearly all models show a significant performance drop from solution planning to next-step planning.The results distinguish offline plan reasoning from dynamic planning during execution.
- Feedback results: Commercial models remain below 65% accuracy on outcome assessment, while error localization and repair are more difficult.Models may identify a nearby erroneous step without determining its specific cause or producing an actionable correction.
- Action results: Commercial models achieve strong and balanced action performance, whereas several open-source models are competitive in calculation but weaker in conceptual understanding and modeling.The action results identify structured mathematical execution as a bottleneck even for models with stronger planning or feedback.
4 Analysis and Discussion
The analysis shows that similar end-to-end performance can conceal substantially different agentic profiles, with weaknesses appearing across planning, action, and feedback. Qualitative cases further reveal recurring failures in multi-step planning and state tracking.
- Similar end-to-end math performance does not necessarily indicate similar agentic abilities.Some math-specialized or problem-solving-focused models perform poorly on core agentic capabilities.
- Planning and Feedback: Planning and feedback scores diverge across models, with commercial models achieving high scores in both while some open models favor one capability.DeepSeek-V3.2 has relatively high feedback performance despite moderate planning, whereas Qwen2.5-Math-72B clusters lower on both dimensions.
- Action Performance: Action profiles vary across six atomic dimensions, and Concept is identified as the primary bottleneck.Commercial and large reasoning-enhanced models form a relatively balanced high-performance envelope, while smaller open models show pronounced weak axes.
- Planning Failure: Case studies identify recurring planning failures in which models collapse multi-step reasoning into final answers and show weak state tracking.These patterns occur especially among models with strong end-to-end accuracy but weaker agentic profiles.
5 Conclusion
AgenticMathBench evaluates mathematical agentic capabilities by decomposing reasoning into atomic capabilities aligned with planning, action, and feedback. Its results show substantial differences in agentic proficiency among models with similar final-answer accuracy, supporting process-level evaluation.
- AgenticMathBench aligns a structured taxonomy of mathematical atomic capabilities with planning, action, and feedback.The benchmark evaluates how models plan, execute, reflect, and repair solutions in textual and multimodal contexts.
- Models with similar final-answer accuracy can exhibit substantial disparities in agentic proficiency.The reported disparities motivate evaluating the reasoning process rather than only the final answer.
Limitations
The framework does not cover every form of mathematical agentic reasoning and has scope limitations in multimodal coverage and cost accounting. Future work targets longer-horizon, more visual, and cost-aware evaluation.
- The taxonomy and task instantiations may not cover all mathematical agentic reasoning, especially long-horizon memory and iterative learning.The authors plan explicit memory perturbation tests and tool-interaction loops for longer-horizon settings.
- The trajectory-based multimodal subset is smaller than the text-only subset because high-difficulty diagram problems are scarce.Expanding diagram-based and visual-reasoning trajectories is identified as future work.
- AMB reports capability scores without fine-grained token and latency accounting because it measures intrinsic capability rather than a deployed agent system.Cost-aware evaluation combining capability gains with inference budgets remains an extension.
Ethics Statement
The study uses publicly available and open-source datasets under their original licenses and terms of use. It evaluates mathematical reasoning behaviors without sensitive personal data.
- All datasets are collected from publicly available and open-source resources and used under their original licenses and terms.
- The benchmark evaluates mathematical reasoning behaviors and does not involve sensitive personal data.The authors report no potential risks.
LLM Usage Statement
The section situates mathematical agentic reasoning within prior benchmark and agent-system research while outlining the taxonomy design process.
- Related Work: Existing mathematical benchmarks span elementary, synthetic, and competition-level datasets but commonly report aggregate accuracy.These sources include DeepMind Mathematics, GSM8K, MATH, OlympiadBench, and related collections.
- Related Work: Agentic mathematical systems commonly use tools, memory, multi-agent structures, or explicit Planner–Reasoner–Executor decompositions.Representative systems include ToRA, MathChat, and MathAgent-PRER.
- Taxonomy Design: The taxonomy distinguishes mathematical data-domain classifications from classifications based on reasoning abilities and problem-solving patterns.The authors argue that ability-focused classifications are closer to agentic capabilities.
- Taxonomy Design: Existing datasets are mapped to the proposed atomic capabilities, while rarely studied capabilities are removed and others are merged or decomposed.The process aims to focus on frequently occurring core capabilities.
- Taxonomy Design: Mathematics experts iteratively refine the taxonomy for diversity, near-completeness, relative independence, evaluability, and agent alignment.The consultation involved three mathematics PhDs and two senior mathematics instructors.
C.2 Curated Atomic Abilities from Existing Benchmarks
AgenticMathBench curates atomic abilities from existing mathematical datasets and constructs composite problems from competition benchmarks, using filtering and annotation procedures to support systematic evaluation.
- Atomic Abilities: Seven of nine atomic abilities are curated from existing datasets and converted into a unified schema without LLM-based rewriting.The schema contains question, answer, and minimal source, split, and ability metadata.
- Atomic Abilities: The curated sources cover handwritten expressions, mathematical statements and proofs, symbolic and numeric problems, geometry, and Lean formalization.Examples include CROHME, NaturalProofs, FormalGeo, and Lean-based corpora.
- Atomic Abilities: Deductive and inductive reasoning and mathematics modeling are generated through LLM-based pipelines because their source problems require substantial restructuring.Their construction procedures are described separately in the appendix.
- Composite Problems: Composite problems come from recent Olympiad and competition benchmarks and require a non-trivial combination of several atomic abilities.The source problems are deduplicated and filtered for mathematical relevance and coherent structure.
- Composite Problems: Each retained composite problem receives a multi-label vector over nine abilities, with disagreements resolved through annotator discussion.Annotators label abilities genuinely required for a complete solution rather than merely mentioned.
- Coverage: Table 7 characterizes benchmark coverage qualitatively using ✓, ◦, and × for primary, secondary, and rare or absent ability requirements.These labels describe source diversity and are not used directly in evaluation.
- Trajectory Filtering: The trajectory pipeline retains 217 pure-text and 20 multimodal trajectories after correctness, human reasoning-quality, and diversity filtering.The final dataset keeps trajectories with at least two reasoning steps and at least two atomic abilities.
- Trajectory Filtering: Only 17.3% of the original trajectories were retained after filtering for correctness, logical rigor, and capability diversity.The low retention rate is described as deliberate for advanced competition-level problems.
D Additional Details of Data Statistics
The section documents task volumes, benchmark coverage, modeling conversion, representative ability examples, and proof-trajectory construction for AgenticMathBench.
- Task Statistics: Planning and feedback task statistics are reported separately for text-only and multimodal settings, including atomic-ability selection and solution planning.Tables 10–12 summarize task volume, coverage, trajectory details, step distributions, and capability usage.
- Mathematics Modeling: The modeling conversion pipeline filters for modelable problems, generates minimal JSON models, and verifies variables, constraints, and objectives against the source question.Verified JSON models are stored as gold answers.
- Ability Examples: Examples instantiate spatial perception as geometric facts, formal language as theorem statements, and deductive reasoning as ordered proof steps.The examples include relations among line lengths, a Hermitian-matrix theorem, and a proof plan.
- Ability Examples: The mathematics-modeling example represents initial and boarded children as variables, relates them through addition, and sets total_children as the objective.The variables are constrained to nonnegative integers.
- Ability Examples: The theorem-application example uses hints about countability, Fσ sets, and De Morgan’s laws to establish that irrationals form a Gδ set.The supplied proof concludes from the countability of the rationals.
- Proof Trajectories: The deductive and inductive reasoning subset converts NaturalProofs theorems and proofs into ordered triples containing an index, proof segment, and plan.A prior proof-checking stage retains only proofs that pass a global sanity check.
- Evaluation: The evaluation pipeline is described as structured, reproducible, and empirically validated.This conclusion is stated after the benchmark’s evaluation procedures.
F.1.2 Verification of LLM-based Evaluation
The verification study compares LLM-based judgments with human annotations across four agentic capabilities and reports strong overall agreement, with a recall-related overestimation caveat.
- Verification Study: The human-aligned study evaluates Solution Planning, Next-step Planning, Error Localization, and Fix Suggestion using a shared rubric.These dimensions represent core competencies for mathematical agentic reasoning.
- Verification Study: The LLM judge shows strong agreement with human annotations across all evaluated dimensions.Agreement is summarized using Accuracy, Precision, Recall, and F1 from confusion matrices.
- Verification Study: The combined evaluation achieves an F1 score of 0.86 against human judgment.The authors describe the multidimensional judge as more fine-grained and comprehensive than single-criterion strategies.
- Caveat: The LLM judge’s high recall bias may slightly overestimate performance by accepting borderline cases.The authors retain this design because it reduces incorrect rejection of valid trajectories when evaluation coverage is important.
- Robustness Check: A GPT-5-mini replication yields Acc. 0.85 vs. 0.86, F1 0.86 vs. 0.85, and 0.96 cross-judge agreement.The replication uses the same sampled outputs and rubric as the default judge comparison.
- Metrics: Several action metrics use an auxiliary LLM judge to semantically match predicted and gold items before computing scores.This matching approach applies to measures such as ConceptSet_F1, RelationF1, and VariableF1.
- Metrics: Symbol recognition is evaluated with expression-level and symbol-level criteria, while calculation uses short algebraic or numeric answers.The corresponding predictions are taken from structured output fields.
- Metrics: Spatial ability evaluation separately scores geometric relation matching and numeric attribute accuracy.Predicted facts and values are compared with gold geometric facts and attributes.
G.2 Results on Multimodal Tasks
The multimodal evaluation covers planning, feedback, and action through dedicated tasks and tables. Qualitative cases show that strong end-to-end accuracy can coexist with process-level weaknesses in planning and feedback.
- Multimodal Planning Tasks: Multimodal tasks evaluate planning through capability selection, solution planning, and next-step planning.
- Multimodal Feedback Tasks: Multimodal feedback tasks assess correctness judgment, error localization, and fix suggestion.
- Multimodal Action Tasks: Multimodal action performance is reported separately in a dedicated evaluation table.
- Failure Analysis: Representative cases reveal shortcut bias, weak state tracking, and inconsistent repairs across models with weaker agentic profiles.Models may collapse multi-step reasoning into final answers, repeat or skip trajectory steps, and offer vague or inconsistent corrections.
- Complete Case Card: In the limit example, the expected diagnosis identifies step 4 as the earliest error and repairs it by differentiating 2x as 2, yielding 1/2.A representative model instead localized the error to step 5, illustrating a causal error-localization failure.
- Data and Annotation Pipeline: Trajectory synthesis and annotation components use controlled rewriting, automated checks, reasoning-quality filtering, and diversity filtering before ground-truth use.
- Cost Discussion: A qualitative cost comparison suggests atomized reasoning can use shorter subgoals than direct long-chain reasoning on difficult AIME25-style problems, without a universal cost guarantee.Direct reasoning can exceed 20K tokens and still be truncated, whereas many atomized solutions use roughly 2–4 steps capped around 4K tokens each.
I.2 Contamination risk.
The benchmark acknowledges contamination risk from public math problems and describes mitigations based on source review, process-level evaluation, metadata release, and future held-out data.
- Contamination risk: Publicly sourced math problems create a known data-contamination concern for the benchmark.
- Mitigations: The authors review more than 150 datasets and prioritize recent competition-level and Olympiad-style benchmarks to mitigate contamination risk.
- Mitigations: Evaluating planning, feedback, and action processes means memorizing a final solution is not sufficient to succeed on AMB.
- Future safeguards: The authors plan to release source and timestamp metadata and periodically refresh the benchmark with newly released contest problems.
- Future safeguards: A strictly held-out split created after evaluated-model release dates is planned for future contamination analysis.