PaperlayerView exact revision on arXiv

Source-linked AI summary

One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning

Armin Dariani, Sifan Wu, Bang Liu, Entao Yang

arXiv:2608.30952v1cs.LGcs.CL

TL;DR

Chemistry tool learning requires selecting tools, supplying accurate arguments, and chaining dependent calls, while existing tree-search systems incur substantial complexity and search cost. This paper trains one policy with supervised warm-up and programmatic-reward reinforcement learning; on ChemToolBench, it improves tool and return F1 over CheMatAgent’s strongest search configurations on both backbones and leads answer Pass Rate on Qwen-2.5-7B.

  • Problem

    Chemistry questions require exact calculations, database lookups, typed tool arguments, and dependent multi-call chains that language-model parameters alone cannot reliably provide.

  • Method

    A single policy interleaves reasoning and live tool execution in one generation, using supervised warm-up followed by GRPO against programmatic rewards derived from validated calling chains.

  • Results

    Tool F1 and Return F1 improve over CheMatAgent’s strongest search configurations on both backbones, with gains of 5.5% and 9.6% on Qwen-2.5-7B and 3.7% and 3.9% on Llama-3.1-8B; answer Pass Rate leads on Qwen-2.5-7B.

  • Takeaways & Limitations

    The reported results show that one policy can match or exceed CheMatAgent’s pipeline on tool, parameter, and return F1 without learned critics or tree search, using one model invocation per question.

  • Takeaways & Limitations

    Evaluation covers one multiple-call comprehensive-chemistry split of ChemToolBench; transfer to other scientific-tool domains, longer chains, and larger models remains untested.

Abstract

from arXiv · show

Chemistry questions often demand exact computation and database lookups that a language model cannot supply from its parameters, so it must reach for external tools. Tool use here is a three-part problem: select the right tool from a large pool, fill it with correctly typed arguments, and chain calls so that each consumes the outputs of the last. CheMatAgent, a previously published system, addresses this with hierarchical evolutionary MCTS: separate policy and execution models searching tool-call trees under two learned critics, one regressed partly onto GPT-assigned scores. We show that a single policy suffices. Our model interleaves reasoning, tool calls, and returns in one left-to-right generation, trained by a supervised warm-up and then outcome-level reinforcement learning against a programmatic reward read directly off the gold call chain, which leaves no learned critic and no judge in the training loop. On ChemToolBench multiple-tool comprehensive chemistry, on both backbones CheMatAgent use, we improve Tool F1 by 5.5% and Return F1 by 9.6% on Qwen-2.5-7B, and by 3.7% and 3.9% on Llama-3.1-8B, compared with their strongest search configuration, at one model invocation per question, against a search whose cost grows with the tree; we also lead answer Pass Rate on Qwen-2.5-7B.

1 Introduction

Chemistry tool use requires selecting specialized tools, filling their arguments, and chaining dependent calls because language-model parameters cannot provide every exact calculation or measured property. The paper replaces CheMatAgent’s costly tree-search machinery with one policy trained by supervised warm-up and programmatic-reward reinforcement learning.

  • Chemistry questions require external tools because model parameters cannot store every measured property or perform all exact calculations.
  • Tool learning is difficult because specialized tools have similar interfaces but differ in assumptions, input requirements, and prediction targets.
  • CheMatAgent separates tool selection and parameter filling across models, then searches tool-call trees under process and outcome critics.
  • The proposed single policy interleaves reasoning, live tool calls, and returned results in one generation, eliminating learned critics and tree search.
  • 5.5% Tool F1 and 9.6% Return F1 improvements on Qwen-2.5-7B and 3.7% and 3.9% on Llama-3.1-8B are reported against the strongest search configurations.

2 Related Work

Prior work established supervised and agentic approaches to general tool learning, chemistry tool augmentation, and verifiable-reward reinforcement learning for reasoning and tool use. This paper combines those directions by using supervised fine-tuning only as a warm-up before online reinforcement learning with live tools.

  • Toolformer and ToolLLM established learned API calling, while API-Bank, Seal-Tools, and ToolACE target tool selection and argument filling.
  • Chemistry agents use expert tools, cheminformatics packages, scientific benchmarks, or chemical pretraining, with CheMatAgent providing the closest prior system.
  • Verifiable-reward reinforcement learning has been extended from reasoning to code execution, search, and tool use, motivating the paper’s online tool-calling training.

3 Task and Benchmark

The task is to answer chemistry questions by selecting typed tools and correctly chaining calls whose later arguments may depend on earlier returns. Experiments use the multi-tool chemistry split of ChemToolBench, with validated gold call chains and unchanged tools.

  • 3.1 Problem Formulation: Each question has a validated chain recording tools, arguments, and returns, which supplies supervised targets and rewards.
  • 3.1 Problem Formulation: Most questions require dependent call sequences rather than one tool call, so agents must select tools and order calls correctly.
  • 3.2 ChemToolBench: ChemToolBench contains chemistry and materials splits with single-tool and multi-tool questions partitioned into training, development, and test sets.
  • 3.2 ChemToolBench: The experiments target 200 test questions from the multiple-tool chemistry split, whose chains contain two to six dependent calls and average 3.15 calls.
  • 3.2 ChemToolBench: The tool pool spans cheminformatics, property and structure prediction, reaction and stoichiometry calculation, and drug-likeness scoring with typed JSON arguments.

4 Method

The method uses one policy for reasoning, tool selection, argument filling, and continuation after live results, then trains it with supervised trajectories and deterministic rewards computed from gold call chains. This removes learned critics and search from the training loop.

  • 4.1 Single-Agent Tool-Use Rollout: A single policy emits reasoning and one typed tool call per turn, reads the server’s returned result, and continues until answering or reaching 16 turns.
  • 4.1 Single-Agent Tool-Use Rollout: The rollout executes real tools during training so later calls can use values returned by earlier calls.
  • 4.2 Supervised Fine-Tuning: Supervised warm-up linearizes gold chains into reasoning, calls, returns, and a final answer before reinforcement learning.
  • 4.3 Programmatic Reward: Programmatic rewards use only the gold chain, are deterministic and reproducible, and avoid learned critics and an LLM judge.
  • 4.3 Programmatic Reward: Tool rewards measure tool-name overlap, while call rewards additionally require normalized argument matches and list/F1 variants penalize extra or repeated calls.
  • 4.3 Programmatic Reward: Hybrid rewards combine tool and answer rewards so the policy is evaluated on both selected calls and reported returned values.

5 Experimental Setup

The evaluation uses CheMatAgent’s benchmark, tools, scoring rules, and shared backbones to isolate training effects. Models are assessed on 200 multi-step chemistry questions with tool-call and answer-level metrics.

  • Dataset and models: The benchmark contains 200 multi-step questions from ChemToolBench’s multiple-tool comprehensive-chemistry test set.Some unreproducible live-service records were removed from training and development data.
  • Dataset and models: Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct match CheMatAgent’s shared backbones, while Qwen3-4B provides a scale check.Within each shared-backbone comparison, only the training method varies.
  • Metrics: Tool F1 measures tool-name matching with list semantics, so duplicated calls count as false positives.The evaluation reimplements CheMatAgent’s scoring code and micro-averages over the test set.
  • Metrics: Parameter scores award partial credit per argument value, while Return compares each produced value with the gold return.A call getting two of three arguments correct receives 2/3 parameter credit.
  • Metrics: Pass Rate is answer-level grading over all 200 questions using CheMatAgent’s prompt and GPT-4o-mini at temperature 0.Missing answers count as No, and Pass Rate is excluded from training rewards.

6 Results

A single policy outperforms or matches CheMatAgent’s search pipeline on tool and return metrics across both shared backbones, while answer-level performance depends on the backbone. Qwen leads Pass Rate, whereas Llama retains a search advantage.

  • Tool selection: 95.76 Tool F1 versus 90.80 for -M3 on Qwen-2.5-7B, and 95.93 versus 92.47 for -M3 on Llama-3.1-8B.The single policy leads Tool precision, recall, and F1 on both shared backbones without inference-time search.
  • Parameters and returns: 89.39/88.39/88.89 Return precision/recall/F1 versus 81.41/80.79/81.10 for -M3 on Qwen-2.5-7B.On Llama-3.1-8B, the corresponding values are 90.06/89.35/89.70 versus 88.64/84.07/86.36 for -M2.
  • Cost and optimization: CheMatAgent’s search uses four trained models and a tree search at every question, whereas the proposed approach uses one model invocation per question.The comparison therefore combines performance with a substantially different inference procedure.
  • Answer Pass Rate: 68.50 Pass Rate versus 67.32 for -M3 on Qwen-2.5-7B, but 64.00 versus 72.30 for -M2 on Llama-3.1-8B.Both single-policy scores exceed the listed chain-of-thought baselines.
  • Cost and optimization: Pass Rate is optimized indirectly for CheMatAgent but is a by-product for the proposed model because its reward never inspects the final answer.The Qwen result remains ahead, while the Llama result shows the tool-level objective is not sufficient everywhere.

7 Analysis

Ablations show that supervised warm-up supplies broad tool-learning signal, while reinforcement learning mainly sharpens precision; reward design prevents recall-driven tool spamming. Live execution also enables self-repair when calls fail.

  • What Each Training Stage Contributes: SFT-only leads every F1 column and Pass Rate over RL-only across all three backbones.Tool F1 margins are 1.43, 0.28, and 8.96; Pass Rate margins are 5.50, 7.00, and 8.50 on Qwen-2.5-7B, Llama-3.1-8B, and Qwen3-4B.
  • What Each Training Stage Contributes: Supervised warm-up provides broader token-level learning, whereas reinforcement learning receives one rollout-level score and primarily improves tool precision at recall’s expense.For Pass Rate, the call-chain reward provides no signal about reporting a complete natural-language answer.
  • Reward Design and Reward Hacking: 98.25 recall from recall-only reward requires 18.33 calls per question, collapsing precision to 19.70 and leaving 97% of episodes without answers.The gold average is 3.15 calls, while half of the excessive calls are repeats.
  • Reward Design and Reward Hacking: A single precision term keeps all three F1 rewards within 0.1 of the gold call count, rarely repeats, and always terminates.The resulting Tool F1 gap spans 32.82 to 95.38, showing the term’s central role in avoiding reward hacking.
  • Reward Design and Reward Hacking: Exact-match return scoring supplies noise: only 57% of numeric returns match exactly, although 92% are reported within 1%, making RF1_hyb least stable.Adding answer scoring does not help, so RF1_call is used because it evaluates parameter and return columns.
  • Self-Repair from Execution Feedback: Live execution turns misnamed or wrongly typed calls into error feedback, allowing the policy to correct and re-issue them rather than propagate failure.This self-repair cannot come from gold-chain supervision because gold chains contain only successful calls.

8 Conclusion

The paper argues that chemistry tool-learning does not require CheMatAgent’s multi-part search machinery. A single policy with supervised warm-up, GRPO, and programmatic rewards matches or exceeds the prior pipeline on tool-learning metrics at one invocation per question.

  • Conclusion: A single policy interleaving reasoning and tool calls matches or exceeds CheMatAgent’s HE-MCTS pipeline on tool, parameter, and return F1.The comparison uses the same backbones and one model invocation per question.
  • Conclusion: Supervised warm-up supplies tool recall, reinforcement learning supplies precision, and list/F1 reward semantics prevent tool-score hacking.The conclusion attributes these roles to the separate training ingredients and reward design.
  • Conclusion: Verifiable programmatic rewards are sufficient at this level without learned reward models or search.This conclusion replaces the prior system’s execution model, critics, and tree search with a single-policy recipe.

Limitations

The evaluation covers one benchmark and one domain, with short call chains and 4–8B backbones. Transfer to other scientific-tool domains, longer horizons, and larger models remains untested.

  • Scope: The study evaluates only ChemToolBench’s multiple-call comprehensive-chemistry split, so in-domain generalisation and transfer to other scientific-tool domains are untested.The authors note that no other known chemistry benchmark pairs questions with multi-call gold chains.
  • Scope: Chains span two to six calls and backbones span 4–8B parameters, leaving longer horizons and larger models unevaluated.

A Refreshing stale gold returns

The benchmark includes live-service tools whose changing outputs can make released gold returns stale, so affected records are removed from training data while the test split remains intact. The evaluation and prompting setup preserves tool signatures, typed calls, returned values, and final-answer grading.

  • Refreshing stale gold returns: 57 of 1623 training and 4 of 200 development questions are dropped because stale gold returns corrupt supervision targets; no test records are dropped.Some tools query live services, so their current outputs no longer match released gold chains.
  • The chemistry tool pool: Ten of 55 chemistry tools query live web services, causing a small number of released gold returns to become unreproducible.The split uses 55 tools across four libraries, invoked through library-qualified names with typed JSON arguments.
  • Prompt and rollout: The prompt requires one-step reasoning, exactly one tool call, waiting for its result, and deciding later calls only after reading returned values.This enforces interleaved execution rather than planning the whole dependent chain in advance.
  • Prompt and rollout: Tool-server outputs are masked from the loss, while model-generated reasoning, calls, and final answers are trained in an identical SFT and RL format.Episodes end at the first Answer: line or after 16 turns; SFT targets linearize gold chains in the same format.
  • Evaluation: Pass Rate is the fraction of Yes verdicts from GPT-4o-mini over 200 test questions, with missing or empty answers counted as No.The grading prompt is reused verbatim from the prior benchmark evaluation.

E A self-repair rollout

The rollouts illustrate live-result dependence and execution-feedback repair. A later call can consume a value returned earlier, while an RL-trained model can correct a malformed call and continue the chain.

  • A complete rollout: The Qwen-2.5-7B SFT→RL rollout matches the gold chain exactly while masking tool-server returns from the loss.Everything outside returned spans is generated by the policy.
  • A complete rollout: The second call uses CID 612298 returned by the first call, because that identifier is absent from the original query.The example therefore demonstrates why later arguments must be formed from live execution results.
  • Self-repair: The RL-only rollout repairs an argument-name error after reading a binding error and re-issues the call correctly.The recovered SMILES is then reused by both later calls, so the initial error could otherwise have cost two more values.
Loading 2608.30952v1…