PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Optimization of Tool Creation and Use for Large Language Model Agents

Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee

arXiv:2608.24571v1cs.AIcs.SE

TL;DR

Existing tool-creation systems rely on fixed APIs or decouple tool writers from tool users, leaving tool quality and reusability insufficiently optimized. SMITH jointly trains both skills in one policy with structured rewards and achieves the strongest reported held-out and transfer results, while its tools also benefit other models.

  • Problem

    Existing systems rely on fixed human-designed APIs and separate tool creation from tool use, without explicitly optimizing tool quality or reusability during LLM training.

  • Method

    SMITH jointly trains tool creation and tool use within one policy, using schema-grounded build and use tasks with separate format, execution, judge-quality, and correctness rewards.

  • Results

    79.8 held-out Reasoning-Gym macro accuracy is the best among evaluated methods, with zero-shot transfer to TabMWP-Hard and GQA and tools that generalize across models.

  • Takeaways & Limitations

    Jointly optimizing creation and use makes the tools a policy writes the tools it can reliably invoke and supports reusable, transferable tool-building capabilities.

  • Takeaways & Limitations

    Whether SMITH’s gains persist, saturate, or invert at the 70B+ scale remains open, and the paper does not formally certify adversarial safety or report human evaluations of schema readability.

Abstract

from arXiv · show

Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can invoke. We propose SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a build task (write a tool from a few examples) or a use task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), without any visual or tabular training data. Tools written by our 4B models also lifted the performance of LFM-2.5-350M and Qwen3-30B-A3B under same reasoning tasks.

1. Introduction

Existing tool-augmented LLMs are constrained by fixed human-designed APIs, while tool-creation methods separate the model that writes a tool from the model that uses it. SMITH addresses this gap by jointly training creation and use in one policy with structured rewards, achieving strong held-out and zero-shot results.

  • Fixed human-designed toolsets may be incomplete, poorly matched to new tasks, or unavailable, motivating dynamic tool creation.
  • Existing tool-creation methods do not explicitly optimize tool quality or reusability during LLM training and decouple tool creation from tool use.
  • SMITH jointly trains tool creation and tool use within a single policy using execution accuracy, judge quality, and format consistency rewards.
  • 79.8% macro accuracy on held-out Reasoning-Gym tasks is achieved by the trained 4B model, outperforming inference-time tool-writing frameworks and larger-teacher distillations.
  • SMITH transfers zero-shot to TabMWP-Hard and GQA despite no tabular or visual training data, while its tools generalize across models.

2. SMITH: Schema-grounded multi-task iterative tool honing

SMITH trains one policy to build reusable tools and use pooled tools, making schema quality directly testable through invocation. Its build rewards separately capture formatting, execution, and judge quality, while use rewards answer correctness with turn-efficiency weighting.

  • SMITH trains tool creation and tool use as complementary skills within one shared policy.Build produces Python code and an OpenAI-compatible JSON schema; use operates from the schema without seeing the code.
  • Task formulation: Build tasks infer a general solution from N=4 question-answer examples and express it as both a Python function and JSON schema.
  • Task formulation: Use tasks invoke pooled tools for held-out questions or build a tool from the same examples before invoking it.
  • Evaluation reward: Build evaluation rewards successful tool-mediated answers on hidden test questions, with a periodically refreshed evaluator providing a stable target.
  • Format reward: Format reward requires exactly one mutually consistent Python block and JSON block, while invalid pairs terminate the rollout with zeroed reward axes.
  • Judge reward: Judge reward scores code correctness, schema quality, and overall quality, with signature mismatches halving the score and syntax errors receiving a fixed negative reward.
  • Use-task reward: Use tasks run for up to T=5 turns and scale final-answer correctness by an efficiency multiplier with η_min=0.3 and η_mid=0.7.
  • Joint policy update: Build and use prompts form equal batch halves, and shared policy parameters receive gradients from both task types in one update.

3. Experiments

Experiments evaluate SMITH on procedural reasoning, cross-task and out-of-domain benchmarks, multiple model backbones, tool consumers, and reward ablations. SMITH generally improves held-out transfer, token efficiency, cross-model tool reuse, and external tool calling.

  • Evaluation setup: Experiments use 13 procedurally generated Reasoning-Gym categories with exact, automatically verifiable answers and difficulty curricula.Evaluation includes RG Seen, RG Unseen, TabMWP-Hard, and GQA, covering increasingly distant transfer settings.
  • Main results: 79.9 RG Unseen accuracy is the highest overall, versus 63.2 for ReTool and 65.8 for LATM-distilled.SMITH also beats scaffolding baselines on held-out tasks, although ReTool leads RG Seen at 92.2 versus 85.2.
  • Main results: 100 average output tokens deliver the strongest aggregate accuracy, roughly 32× fewer than Standard CoT and 6× fewer than ReTool.Input usage is 664 tokens, comparable to LATM and below CRAFT and ReTool.
  • Tools transfer across model scale: 42.9 RG Unseen accuracy with LFM2.5-350M matches the 30B tool writer’s 41.5 when using tools from the RL-trained 4B writer.On training tasks, the 4B writer reaches 21.1 versus 10.4 for the untrained 30B model.
  • Tools transfer across model scale: SMITH-written tools raise the Qwen3-30B-A3B consumer’s overall score from 70.2 to 76.6, with TabMWP-Hard increasing from 0.7 to 38.8.The RL-trained 4B writer improves the larger consumer over the same 30B model writing its own tools across every reported group.
  • Out-of-distribution evaluation: 40.4 TabMWP-Hard accuracy leads the OOD benchmark, while GQA reaches 42.6 and ranks second.These benchmarks were absent from training; the authors note that GPT-4.1 distillation retains an advantage on visual primitives.
  • Out-of-distribution evaluation: 81.7 Qwen3-8B RG Unseen accuracy and 56.7 TabMWP-Hard accuracy improve from 72.2 and 42.4 after SMITH training.Granite-3.3-8B follows the same improvement trend from a weaker starting point.
  • External tool calling: 55.8 BFCL accuracy improves from 43.3 on Qwen3-8B, despite BFCL schemas, traces, and judges being unseen during RL.SMITH also improves BFCL accuracy on Qwen3-4B from 45.1 to 48.6.

4. Limitations and discussions

The paper identifies open boundaries around scale, judge dependence, artifact representation, and multi-tool interaction. It also notes unresolved safety and human-usability evaluation gaps.

  • Scale, judge dependence, and base priors: Whether SMITH’s gains persist, saturate, or invert at 70B+ scale remains an open question.The reported models are at most 8B parameters, while the quality judge has 30B activated parameters.
  • Scale, judge dependence, and base priors: Self-Judge improves held-out performance but degrades out-of-domain GQA, making judge-size dependence unresolved.The authors describe Self-Judge as only a partial probe of judge-size dependence.
  • Scale, judge dependence, and base priors: SMITH does not formally certify that adversarial prompts cannot induce unsafe generated tools.Generated Python executes at every step inside a sandbox, but formal safety certification is absent.
  • Scale, judge dependence, and base priors: The evaluation reports no human assessment of schema readability or developer-facing reusability.This leaves practical usability of generated schemas unmeasured.
  • Open Questions: The work evaluates one Python function plus a JSON schema, not richer multi-file skills or server-backed artifacts.Supporting scripts, references, templates, resources, or MCP-style interfaces would require extending the output representation and validation pipeline.
  • Open Questions: SMITH never produced parallel tool calls or multiple tools in one turn, leaving common deep-research workflows for future work.The authors attribute this to the base model’s single-tool tendency and reward design, while noting that extension may be natural.

5. Conclusion

The paper introduces SMITH as a single-policy reinforcement learning framework that jointly trains reusable tool creation and tool use. Across the reported evaluations, the 4B model achieves the highest held-out Reasoning-Gym accuracy, leads TabMWP-Hard, and transfers across models and sizes.

  • 5. Conclusion: SMITH jointly trains a single language model to create and use reusable tools, directly optimizing the feedback loop between writing and execution.The framework is presented as reinforcement learning that closes the loop between tool writer and tool user.
  • 5. Conclusion: 79.8 held-out Reasoning-Gym accuracy is achieved by the 4B model, the highest among evaluated methods.The model also leads TabMWP-Hard and writes tools that transfer to a 350M student and match tools from a much larger model.
  • 5. Conclusion: SMITH lifts Qwen3-8B and Granite-3.3-8B without modification, indicating transfer of the coupled creation-and-use recipe across backbones.The conclusion reports this as evidence that the training recipe is not limited to one backbone.

Appendices

The appendices document SMITH’s joint tool-creation and tool-use setup, then analyze a training collapse caused by inconsistent structural penalties and naming drift. They derive fixes emphasizing aligned verifier–judge rewards and explicit structural checks.

  • System and Related Work: Existing systems rely on fixed human-curated tools or inference-time prompting, whereas SMITH makes reusable tool synthesis and invocation explicit RL objectives.Prior systems include LATM, CRAFT, Trove, and KTCE; ReTool improves use reliability without addressing creation.
  • System and Related Work: SMITH jointly trains tool creation and tool use, producing a reusable Python function plus an OpenAI-compatible JSON schema for invocation.Build tasks synthesize both artifacts; use tasks expose only the schema and require tool invocation on held-out questions.
  • Observed Failure: At step 6, 46 of 238 build states failed the first structural check, versus 5 of 197 at step 5, with failures split between function-name and parameter mismatches.The resulting evaluation-reward collapse followed directly from these step1 failures receiving zero reward.
  • Observed Failure: Items reaching evaluation scored 5.26/8 at step 6 versus 5.44/8 at step 5, showing evaluator quality changed only mildly while structural failures drove the collapse.No evaluator connection errors or exceptions were observed.
  • Observed Failure: Naming drift occurred when the schema referenced a function absent from the generated code, while verifier and judge penalties treated the same structural failure inconsistently.The judge rewarded elaborate multi-function code even when the schema-named callable was missing, reinforcing the failure through the gradient update.
  • Lessons and Fixes: The corrective principle is to make judge penalties match verifier-enforced hard failures and explicitly instruct the judge to check schema–code alignment.The revised prompt requires the schema name to appear verbatim as a top-level definition and includes scored counterexamples.

B.4.4. Lesson 4: Log Structural Sub-Metrics as First-Class Signals

Aggregate reward can conceal degradation in a specific structural constraint, so training should expose and monitor violation rates independently. The section illustrates this with naming drift and recommends threshold-based alerts.

  • Failure Detection: At step 5, aggregate metrics appeared healthy even though step 6 produced a sudden structural failure; only sub-metrics revealed the emerging problem.Function-name mismatch declined from 23.4% at step 1 to 0.0% at step 5, while parameter mismatch remained 1–3%.
  • Failure Detection: The monitoring surface should include step1_fail, fn_mismatch, and param_mismatch as separate time-series metrics.These respectively capture early structural failure, schema-name absence, and schema–signature parameter disagreement.
  • Failure Detection: A step1_fail alert above 10% would have detected the regression at step 6 before eval_reward collapsed further.Step1-failed rollouts terminate before evaluator and judge scoring, making step1_fail the only direct signal for them.
  • Failure Mechanism: The policy learned that multi-function helper structures correlated with high evaluation reward, then generalized that style beyond the schema-naming constraint.This reward hacking was visible in the fn_mismatch trajectory rather than the aggregate reward trajectory.
  • Design Principle: Structural constraints should be hard-coded in verifier rewards, while each violation rate receives independent monitoring and threshold-based alerts.The recommended response is a targeted fix rather than a global hyper-parameter change.
  • Judge Evaluation: The judge prompt evaluates tools through observable structural and coverage signals, including code structure, task coverage, and schema–code comparison.The examples warn against rewarding polished structure when the implementation cannot handle the task.

F. Full Easy-to-Hard Difficulty Split for All 13 Training Tasks

The easy-to-hard protocol trains tools from easier procedural examples and evaluates them on the hardest available instances across 13 diverse task categories. The appendix specifies dataset sizes and the single-shot build-then-use procedure.

  • Difficulty Split: For every task, induction uses Band 2 examples while evaluation uses the hardest available difficulty band, preventing tools from succeeding through memorization alone.Higher band numbers indicate harder procedural-generator instances.
  • Dataset Construction: Training uses 500 Band 2 instances per task, with an additional 500 Band 3 instances for tasks needing a larger easy-to-hard gap.The evaluation set samples K=16 instances from the hardest band during rollout reward computation.
  • Task Coverage: The 13 categories cover arithmetic, algorithms, algebra, games, and logic, exposing the policy to varied reasoning patterns.Categories include bitwise arithmetic, LCM, polynomial equations, Tower of Hanoi, and knights-and-knaves.
  • Evaluation Procedure: Evaluation performs one build pass from in-context examples, discards failed tools without retry, and applies the resulting tool to every test instance.Use tasks allow up to T=5 tool calls before a final answer.
  • Evaluation Procedure: The build context is drawn from each benchmark’s training split, with four examples for TabMWP-Hard and ten question–answer pairs for GQA.GQA provides no images during building; the model must infer a strategy through the available visual primitives.

G.3. Protocol Differences from the RG Training Loop

Out-of-domain evaluation removes the training loop’s easy-to-hard curriculum, pooled tools, and evaluation reward, using fresh single-shot builds from benchmark-specific examples. GQA additionally constrains the model to compose fixed visual primitives rather than implement vision code.

  • Protocol Differences: OOD evaluation draws induction examples and test instances from the same distribution, unlike Reasoning-Gym’s easy-to-hard separation.This removes the curriculum gap between build examples and evaluation instances.
  • Protocol Differences: No pre-built Tool Pool is injected at OOD evaluation; every benchmark run performs a fresh build from its in-context examples.This tests tool-writing transfer without reusing training-distribution tools.
  • Protocol Differences: OOD evaluation computes no r_eval and uses only final test-split accuracy as its performance signal.The training-time held-out Reasoning-Gym reward is therefore absent from this evaluation loop.
  • GQA Setup: GQA supplies three fixed visual primitives—locate_objects, visual_qa, and crop_region—that the generated tool composes through a local API.SMITH does not implement low-level vision code during evaluation.
  • GQA Setup: GQA tools use a question-only interface and access each image through the preloaded global IMAGE variable encoded as a base-64 JPEG.The generated tool receives question: str and returns a plain string.
  • GQA Setup: The GQA build prompt requires composing the declared primitives, forbids reimplementing them, and provides ten training-split question–answer examples without images.The larger context supports inference across more diverse visual question types.

H.4. Tool Selection and Evaluation

The GQA adaptation generates multiple candidate tools, selects the best on validation accuracy, and evaluates it on the test split. ReTool is adapted with visual primitives and per-question code execution rather than reusable schemas.

  • Tool selection: 10 candidate Python tools are generated for GQA and evaluated on 100 disjoint validation questions.The candidates are built from 10 reference question-answer pairs sampled from GQA training data.
  • Tool selection: The candidate with highest validation accuracy is selected and applied to the full GQA test split.Failed builds receive zero validation accuracy and are normally excluded from selection.
  • Evaluation: GQA uses exact string match after lowercasing and whitespace stripping, without an LLM equivalence judge.This rule-based evaluation is used for short answers such as “yes”, “blue”, and “3”.
  • ReTool comparison: ReTool generates ephemeral code per question with up to 10 execution rounds, rather than performing a reusable tool build pass.Each question triggers a fresh code-generation episode and terminates on a final answer or exhausted turn budget.
  • ReTool comparison: ReTool’s GQA adaptation exposes locate_objects, visual_qa, and crop_region, prepending their implementations so sandbox calls can use vision.These additions let ReTool incorporate visual functions directly into its reasoning chains.

I.3. Key Distinction from SMITH

SMITH and ReTool both use execution feedback, but differ in reusability and training signal. ReTool favors seen-task performance through distillation, whereas SMITH’s advantage appears on held-out transfer.

  • Reusability: SMITH stores callable JSON schemas in a Tool Pool for reuse, while ReTool discards generated code after each question.The schema provides a structured interface that can be invoked across multiple questions without rebuilding the tool.
  • Training signal: SMITH trains end-to-end from verifiable rewards, whereas ReTool uses behavioral cloning from a 32B oracle.SMITH therefore does not require a strong oracle at training time.
  • Results: 92.0 versus 86.6 on RG (Seen) shows ReTool’s higher in-distribution accuracy under 32B distillation.The comparison reflects the advantage of the distillation oracle on seen task families.
  • ReTool training: ReTool’s distilled model is trained on GPT-4.1 tool-making and successful tool-use trajectories, with distractor schemas added to tool-use examples.The final dataset contains tool-making and distractor-augmented tool-use conversations for all 13 training categories.
  • Experimental control: The LATM comparison uses full-parameter updates versus SMITH’s LoRA configuration, so it primarily tests training signal quality rather than adapter capacity.LATM is evaluated autonomously at test time using four in-context examples.

J.4. Comparison to SMITH

SMITH replaces offline or retrieval-centered tool-library optimization with RL that directly improves one policy’s tool creation and use. Its retrieval-free design writes needed tools from scratch.

  • Training objective: LATM imitates GPT-4.1 outputs, whereas SMITH conditions every update on whether generated tools answer held-out questions correctly.SMITH’s training signal is verifiable execution reward rather than oracle demonstrations.
  • Transfer results: 79.8 versus 66.4 on RG Unseen marks SMITH’s transfer advantage over LATM, while GQA reports 42.6 versus 56.0†.LATM remains strong in-distribution, reaching 83.3 on RG Seen.
  • Transfer results: SMITH reaches its GQA score through self-supervised tool creation without visual oracle data, unlike GPT-4.1 distillation.The reported GQA distillation gap reflects GPT-4.1’s native visual understanding and generated visual primitives.
  • Baseline designs: KTCE evolves tools offline and retrieves them online, while CRAFT builds an offline library and TroVE grows a library sequentially during testing.Their tool-creation timelines differ from SMITH’s jointly trained policy.
  • TroVE limitation: Problem order affects TroVE because later problems can use tools created for earlier ones.TroVE runs three parallel modes per problem and prunes low-frequency tools every 500 problems.
  • Optimization target: SMITH’s optimization is RL training itself, directly improving the policy that generates tools instead of refining individual tools afterward.KTCE uses delete/modify/add iterations, TroVE uses frequency-based forgetting, and CRAFT has no post-creation optimization.
  • Retrieval: SMITH eliminates retrieval by writing the needed tool from scratch, bypassing the index or library lookup required by KTCE and CRAFT.TroVE instead injects its entire trimmed library into the prompt.

L.1. KTCE: Three Compounding Failure Modes

KTCE’s reported profile is confounded by solver bypasses, degenerate tools, unstable endpoints, and incomplete token records. These issues can inflate some scores while collapsing computation-heavy tasks and understating resource use.

  • Programmatic solver bypass: KTCE reaches 100% on several tasks through a hand-coded dispatcher without invoking its generated tools.This bypass masks failures in the evolutionary tool-generation pipeline for those categories.
  • Programmatic solver bypass: Cryptarithm, knights_knaves, and polynomial_equations fall to 5.7%, 4.8%, and 1.1% when the dispatcher is incomplete or absent.GQA reaches 0% because the dispatcher lacks visual question-answering implementation.
  • Stub tool generation: Approximately half of KTCE tasks receive stub tools whose bodies simply return an empty string.Execution can succeed while the resulting answer remains unusable.
  • Stub tool generation: 96.7% on syllogism and 96.2% on gcd contrast with 0% on ab and group_anagrams when textual reasoning suffices but working tools are needed for computation.Base_conversion also scores 0% in the cited breakdown.
  • Endpoint instability: The ab task has real LLM responses for only 64 of 210 entries, illustrating endpoint instability that also corrupts aggregate result files.Failed calls can return no generated code or token-usage record.
  • Token accounting: 12 of 20 TroVE tasks lack stored Phase 2 token usage, and reconstructing prompts with the initial toolbox severely underestimates their size.The erroneous reconstruction produced an average of 691 tokens from a prompt of roughly 179 characters.
  • Token accounting: 829 tokens is a lower bound for TroVE’s macro-average because it uses only the 8 tasks with real API-recorded usage.The excluded tasks may have larger prompts due to larger learned libraries.

M. TabMWP-Hard: Dataset Construction and Augmentation

TabMWP-Hard is constructed to expose weaknesses hidden by clean, small tables by combining large-scale distractors, near-miss entries, and richer table schemas. The augmentation pipeline preserves answerability while making lookup, column selection, and exact matching substantially more demanding.

  • Motivation: Standard CoT reaches 96.8% on original TabMWP, motivating TabMWP-Hard because clean tables can make simple lookup resemble tabular reasoning.The harder set is derived from the same problems but targets models that merely identify an entity and read its value.
  • Table selection: Three table types—Financial Ledger, Price List, and Two-Way—are selected for full augmentation, while bounded or rigid types are excluded.Stem-and-Leaf and Function Table formats leave too little freedom for confusing distractors.
  • Augmentation pipeline: Each selected entry undergoes four sequential stages: distractor injection, near-miss insertion, LLM column augmentation, and final validation.The pipeline also checks surviving entries for key collisions before writing the dataset.
  • Distractor injection: Up to 2,000 compatible distractor rows are randomly inserted, burying target rows at arbitrary positions.Generated values remain within each table’s observed type and range.
  • Near-miss insertion: Near-miss rows place similar but distinct keys beside targets, forcing models to distinguish names such as “Eve” from “Ev” or “Elliot.”For Price List entries, semantically related shell names are inserted near the two target items.
  • Column augmentation: LLM column augmentation adds plausible irrelevant fields and, for headerless Price Lists, infers existing column names before constructing a header.The resulting examples include a 7-column, 308-row table and a 6-column, 316-row table.
  • Resulting challenge: The final tasks require locating targets among hundreds of rows, selecting the relevant value, and resisting adjacent distractors with plausible alternatives.This applies simultaneously to noisy philanthropic-donation and shell-price examples.

O.2. Results

The results section evaluates TroVE prompt variants using paired exact-match comparisons, while also documenting grading and data-quality constraints. Across statistically meaningful comparisons, no tested variant significantly outperforms v1, and several changes regress on affected tasks.

  • Evaluation setup: Table 17 compares four TroVE prompt versions on eight reasoning tasks using paired exact McNemar tests against v1.The evaluation uses a frozen toolbox and strict exact-match grading.
  • Ceiling and flat tasks: Three tasks are at or near ceiling across versions, while knights_knaves remains flat at 50.5–52.9%.count_bits and isomorphic_string stay at 100.0%, and chinese_theorem ranges from 96–100%.
  • Task-specific regressions: On caesar_cipher, v3 falls from 59.2% to 49.5% and v4 to 51.5%, with both regressions statistically significant against v1.The v3 and v4 results do not differ significantly from each other.
  • Task-specific regressions: On polynomial_equations, only v3 significantly regresses, declining from 37.9% to 31.4%, while v4 is statistically indistinguishable from v1.The comparison suggests punctuation changes contributed more to regressions on the polynomial tasks.
  • Grading caveats: Exact-string grading can confound reasoning with formatting: numeric re-grading makes bitwise_arithmetic near-ceiling and removes significant version differences.Under numeric grading, accuracies range from 97.5% to 100.0%, with p≥0.18 for every comparison against v1.
  • Data-quality and scale caveats: Two tasks are excluded from significance testing because repeated question text collapses their effective paired comparisons, while GSM8K shows no significant version advantage.On GSM8K, v1 reaches 91.1%, versus 90.2% for v2 and 90.8% for v3.
  • Overall takeaway: Across statistically meaningful comparisons, neither v2 nor its isolated changes significantly outperforms v1, supporting v1 for the paper’s non-ablation TroVE results.The authors therefore argue that the SMITH–TroVE gap is not an artifact of an under-tuned TroVE prompt.
Loading 2608.24571v1…