AlphaApollo: A System for Deep Agentic Reasoning

1 Introduction

• Foundation models face limited capacity for complex tasks, while prompting and post-training rely on base-model priors and test-time refinement can lack trustworthy verification.The introduction also identifies under-explored scalable parallel evolution and the need for memory to support long-horizon evolution.Source: §1 · p. 1
• AlphaApollo coordinates models and tools as a self-evolving system aimed at overcoming these reasoning bottlenecks.The system is organized around three features intended to transcend single-model capacity limits.Source: §1 · p. 2
• Multi-turn agentic reasoning uses structured model actions, tool feedback, and accumulated interaction history as dynamic memory.Actions may be tool calls or answers, followed by tool execution and environment feedback.Source: §1 · p. 2
• Turn-level agentic learning optimizes reasoning and tool invocation while separating model actions from tool responses to stabilize training and improve tool-use decisions.The decisions include what to call, what to query, and when to stop.Source: §1 · p. 2
• Multi-round agentic evolution refines candidate solutions through a propose-judge-update loop with tool-assisted checks and selective memory, and can coordinate multiple models.The process refines solutions over multiple rounds.Source: §1 · p. 2
• Across seven math benchmarks, Avg@32 rises from 1.07% to 9.64% for Qwen2.5-1.5B and from 8.77% to 20.35% for Qwen2.5-7B after multi-turn RL; evolution adds further gains.Tool-call success exceeds 85% across datasets.Source: §1 · p. 2

2.1 Multi-turn Agentic Reasoning

• The model and environment exchange actions and feedback across turns, with each turn’s prompt incorporating earlier interaction history until an answer is produced or the turn limit is reached.The environment hosts tools, parses outputs, executes calls, and returns tool feedback; computational and retrieval tools are provided.Source: §2.1 · pp. 2–3

2.2 Multi-turn Agentic Learning

• Turn-level optimization targets model-generated outputs separately from environment feedback, a design intended to support stable learning on multi-turn interactions.GRPO uses a reward based on the answer proposed in the final turn, with a baseline and normalization across trajectories.Source: §2.2 · pp. 4–5

2.3 Multi-round Agentic Evolution

• At test time, specialized agents generate, condense, verify, and summarize candidate solutions in repeated rounds, with stored judgments conditioning later rounds.The solver produces a multi-turn trajectory; the abstractor condenses it, the evaluator generates verifications, and the summarizer turns them into guidance for later solving.Source: §2.3 · p. 6
• Long-term memory stores each round’s solution and judgment, then retrieves top-scoring entries prioritizing correct and shorter solutions.Retrieval is intended to promote promising strategies and prevent recurrent errors.Source: §2.3 · p. 6
• Parallel evolution lets heterogeneous solvers share high-quality solutions through synchronized long-term memory.Worker threads can use different models or sampling temperatures, and shared memory makes committed solutions available to other solvers.Source: §2.3 · p. 6

3 Experiments

• Experiments cover Qwen2.5 models from 1.5B to 14B across AIME24, AIME25, CMIMC, HMMT February and November, BRUMO, and SMT, with up to four interaction rounds.Source: §3 · p. 6
• Tool-enabled reasoning improves Avg@32 from 4.66% to 4.72% on Qwen2.5-3B-Instruct and from 10.82% to 13.49% on Qwen2.5-14B-Instruct.Tool-call success exceeds 85% across evaluated datasets.Source: Table 1 · p. 7, §3 · p. 7, Figure 5 · p. 8
• Average accuracy gains over no-training baselines are +8.57 on Qwen2.5-1.5B-Instruct, +8.33 on Qwen2.5-3B-Instruct, and +11.58 on Qwen2.5-7B-Instruct.For Qwen2.5-3B-Instruct on MATH-LightEval, accuracy rises from 4.72% to 13.35%.Source: Table 2 · p. 7, §3 · p. 8
• 16.53% to 21.08% (+4.55) on Qwen2.5-14B-Instruct average accuracy with evolution; Qwen2.5-3B-Instruct gains +2.43 on average.Accuracy improves across successive evolution rounds, with scale-dependent gaps.Source: Table 3 · p. 7, §3 · p. 8, Figure 9 · p. 9
• Training and test accuracy grow steadily; full-parameter training learns faster, reduces entropy loss, and finishes above LoRA.The reported comparison is for Qwen2.5-14B-Instruct on MATH-LightEval.Source: §3 · p. 8, Figure 6 · p. 8, Figure 7 · p. 9
• Case studies describe decomposition, correction, verification, and backtracking as reasoning behaviors exhibited under AlphaApollo.The behaviors are illustrated in Figure 10.Source: §3 · pp. 8–9, Figure 10 · p. 10

4 Conclusion

• AlphaApollo combines structured multi-turn tool use, turn-level learning, and multi-round evolution with tool-assisted verification and long-horizon memory.Source: §4 · p. 10

A Related Work

• The related-work review organizes prior research around tool-integrated reasoning, multi-model reasoning, and test-time iteration.Source: A Related Work · p. 17

A.1 Tool-integrated Reasoning

• Prior tool-integrated work spans external support for knowledge and arithmetic, dynamic tool-using agents, and tool-learning, where long-horizon optimization remains challenging.Source: §A.1 · pp. 17–18

A.2 Multi-model Reasoning

• Prior multi-model reasoning includes collaboration, debate, and joint fine-tuning, including role-specialized optimization and memory-based online reinforcement learning.Source: §A.2 · p. 18

A.3 Test-time Iteration

• Test-time iteration ranges from independent parallel sampling through sequential refinement to mixed strategies such as MCTS expansion and outcome simulation.Source: §A.3 · p. 19

B.1 The Computational Module

• The computational module integrates a Python interpreter with external Python libraries to address complex reasoning tasks.Figure 12 schematically illustrates the retrieval module’s information-retrieval process, with purple components representing the Retrieval Server.Source: §B.1 · p. 20, Figure 12 · p. 21
• The computational module includes SymPy for symbolic mathematics, NumPy for vectorized numerical computing, and SciPy for advanced scientific and engineering algorithms.Examples include closed-form equation solving, matrix operations, and numerical optimization.Source: §B.1 · p. 20
• Python execution runs in an individual subprocess isolated from AlphaApollo’s main process, and users can extend the shared environment by installing libraries.The computational module runs model-generated Python code from a temporary file.Source: §B.1 · p. 20
• Rule-based correction fixes identifiable indentation and syntax errors, while retrieval can guide responses to some library-related runtime errors.The module can correct indentation and remove extraneous Markdown wrappers; ValueError, TypeError, AttributeError, and ImportError may trigger retrieval for guidance.Source: §B.1 · pp. 20–21

B.2 The Retrieval Module

• The retrieval module rewrites queries, retrieves relevant library documentation, and summarizes it into callable functions, required arguments, and minimal working examples.Its stated aim is to ground generated code in reliable documentation and improve function invocation.Source: §B.2 · pp. 21–22

B.3 Experimental Settings

• The baseline uses up to 4 rounds of Python execution and local retrieval, temperature 0.6, and history capped at 8, with no additional learning or verification.Each question is evaluated in a single pass.Source: §B.3 · p. 22

B.4 Prompts

• The first-turn prompt asks the model to solve step by step, choose tools, and present its final answer in a boxed format.Subsequent-turn prompts incorporate memory from previous turns while retaining those instructions.Source: §B.4 · p. 22

• Prior trajectory-level RL and SFT mask external-tool responses, optimizing model-generated content while learning from the agentic reasoning trajectory.The passage describes trajectory-level masking as a way to prevent the policy from learning tool responses.Source: C Agentic Learning: Implementation… · pp. 22–23
• The first-turn prompt asks the model to solve step by step, select tools, and provide a boxed final answer; later turns also use prior-turn memory.These prompt instructions are shown for the first and subsequent turns.Source: Figure 13 · p. 23, Figure 14 · p. 23
• Long-horizon agentic reasoning can still destabilize training despite masking tool responses, with low-probability tokens potentially contributing to reward collapse.The cited studies report reward collapse in agentic reinforcement learning.Source: C Agentic Learning: Implementation… · p. 24

C.1 Experimental Settings

• Training uses Lighteval-Math, LIMR, and DeepScaleR in separate experiments, with learning rate 1e-6, batch size 128, group size 8, and temperature 0.4.The temperature is set for stable performance monitoring during reinforcement learning.Source: §C.1 · p. 24

C.2 Prompts

• The experiments use the same prompts as Appendix B.4, shown in Figures 13 and 14.Source: §C.2 · p. 24

D.1 Experimental Settings

• Agentic evolution uses two models without training, runs for 10 rounds, and repeats verification five times for a majority judgment.The policy model uses temperature 0.7, the verifier 0.4, and both have an 8k-token limit.Source: §D.1 · p. 24, Table 4 · p. 24

D.2 Prompts

• Subsequent Solver turns use long-term memory of prior attempts alongside local interaction context to advance solutions.The Abstractor compresses logs into logical checkpoints, filtering computational noise, syntax errors, and backtracking.Source: §D.2 · pp. 24–25, Figure 16 · p. 26, Figure 17 · p. 27
• The Evaluator checks candidate solutions against the problem, may use tools to validate intermediate steps, and issues a binary correctness judgment.Later verification turns include interaction history.Source: §D.2 · p. 25, Figure 18 · p. 28, Figure 19 · p. 29
• When evaluators agree, the Summarizer combines their arguments into one concise judgment while preserving the consensus.It synthesizes arguments and removes redundancy.Source: §D.2 · p. 25, Figure 20 · p. 30
• Table 5 distinguishes tool use without evolution (w/o Evo) from tool-assisted agentic evolution (+Evo); bold marks the better result.Source: Table 5 · p. 31

E.1 Agentic evolution

• 26.81% to 47.10% average accuracy: Qwen3-4B-Instruct improves with agentic evolution versus tool use without evolution.Source: §E.1 · p. 31
• +36.70 on AIME24 and +29.18 on AIME25 are the largest reported Qwen3-4B-Instruct gains.HMMT25 Feb gains +27.50, BRUMO25 +27.53, and HMMT25 Nov +19.23.Source: §E.1 · p. 31

E Additional Experimental Results

• The authors report that Qwen3 results further demonstrate robust and consistent improvement in mathematical reasoning with tools from agentic evolution.Source: §E.1 · p. 31

F.1 Python Code Errors

• Rule-based correction removes markdown code fences that cause SyntaxError and fixes unexpected indentation.Source: §F.1 · p. 31, Figure 21 · p. 32, Figure 22 · p. 32
• For errors needing model-based correction, feedback guides checks for syntax, undefined names or packages, types, values, and imports.Source: Figure 23 · p. 33, Figure 24 · p. 33, Figure 26 · p. 35

F.2 Cognitive Behaviors

• In the lottery example, the response decomposes the conditional-probability problem into total combinations and counts of outcomes with two, three, or four matches.It calls Python's combination function to compute the total number of four-number selections as 210.Source: §F.2 · p. 40
• For the digit-divisibility problem, the model formulates modular constraints and tries a descending Python search, then revisits its approach after the code returns None.Source: §F.2 · p. 42
• Another response rewrites a symmetric expression using a+b+c=300 and proceeds toward a Python search for valid integer triples.Source: §F.2 · p. 45