Joint Optimization of Tool Creation and Use for Large Language Model Agents

• Three signals—execution accuracy, LLM-as-judge quality, and format consistency—separate schema and implementation failures, while a periodically synchronized judge reduces circular-evaluation risks.The judge evolves alongside the policy without sharing live training weights, supporting evaluation stability.Source: §1 · p. 2
• SMITH jointly trains tool creation and use within one policy, rewarding tools on harder downstream queries to encourage reusable abstractions rather than task-specific shortcuts.The framework is trained on 13 procedural task families from Reasoning-Gym and evaluates in-domain generalization and zero-shot transfer.Source: §1 · p. 2
• 79.8% macro accuracy makes SMITH the best overall held-out Reasoning-Gym method, surpassing inference-time tool writers, larger-teacher distillations, and a 30B tool writer.SMITH uses substantially fewer decoding tokens than the 30B inference-time tool writer.Source: §1 · p. 2
• +7.6 points on GQA over same-backbone baselines demonstrates zero-shot visual transfer, while SMITH also reaches state-of-the-art performance on TabMWP-Hard without tabular or visual training data.These transfers extend beyond the procedural tasks used for training.Source: §1 · p. 2
• Tools written by the 4B policy let a frozen 350M model match a 30B tool writer, and the recipe consistently improves Qwen3-8B and Granite-3.3-8B.These results indicate transfer of learned tools across models and benefits across backbones.Source: §1 · p. 2

2. SMITH: Schema-grounded multi-task iterative tool honing

• SMITH jointly trains tool creation and tool use: generated Python code and JSON schemas must support later invocation from schema alone.Build tasks infer a reusable strategy from N=4 question-answer examples, while use tasks invoke pooled tools or build one before invocation.Source: §2 · pp. 2–3

2.1. Build task rewards

• Build-task learning uses separate format, execution, and judge-quality signals, with successful tools added to the shared Tool Pool.Format checks require mutually consistent Python and JSON blocks; execution rewards verified answers obtained through successful tool calls, while judge scores code, schema, and overall quality.Source: §2.1 · p. 4

2.2. Use task rewards

• Use-task correctness is efficiency-weighted across up to T=5 turns, and balanced build/use batches update shared policy parameters through three independent reward axes.The use reward preserves a minimum correctness incentive at the turn limit; build and use gradients are accumulated in one update without explicitly combining their advantages.Source: §2.2 · pp. 4–5

3.1. Training tasks

• Training uses 13 procedurally generated, exactly verifiable Reasoning-Gym categories, with easier induction examples and hardest-band evaluation to enforce easy-to-hard algorithmic generalization.The benchmark reports macro-average accuracy on 13 seen categories and 10 entirely withheld task types, plus strengthened TabMWP-Hard and visual GQA.Source: §3.1 · p. 5, §3.2 · pp. 5–6, §3.4 · p. 6

3.3. Models and training configuration

• Experiments use Qwen3-4B-Instruct with LoRA and DAPO, compare same-backbone inference-time and distillation baselines, and include Qwen3-30B-A3B as a scaling probe.The Tool Pool supplies one correct-category tool and two distractors, testing schema identification and invocation under a consistent evaluation protocol.Source: §3.3 · p. 6, §3.5 · pp. 6–7

3.6. Main results

• 79.9 RG (Unseen) accuracy makes SMITH the strongest method, beating ReTool (63.2), LATM-distill (65.8), and every scaffolding baseline, while using 32× fewer output tokens than Standard CoT.SMITH also reaches 85.2 on RG (Seen), although ReTool leads that split at 92.2.Source: §3.6 · p. 7, Table 1 · p. 8

3.7. Tools transfer across model scale

• 42.9 RG (Unseen) with LFM2.5-350M matches Qwen3-30B-A3B’s 41.5, while SMITH-written tools raise a 30B consumer’s overall score from 70.2 to 76.6.The 4B writer also improves LFM2.5-350M from 11.6 to 42.9 and lifts Qwen3-30B-A3B on every shared task group.Source: Table 2 · p. 8, §3.7 · pp. 8–9, Table 3 · p. 9

3.8. Out-of-distribution evaluation

• 40.4 on TabMWP-Hard leads the closest baseline TroVE (36.4), while 42.6 on GQA ranks second behind GPT-4.1-distilled LATM (56.0) without visual or tabular training data.SMITH outperforms every scaffolding baseline on both out-of-domain benchmarks and is the only same-backbone 4B method without oracle distillation to lead either column.Source: Table 4 · p. 9, §3.8 · p. 9

3.9. Scaling across backbones

• 81.7 RG (Unseen) and 56.7 TabMWP-Hard accuracy show SMITH improves Qwen3-8B, while Granite-3.3-8B also improves across RG and GQA.Self-Judging further raises Qwen3-8B held-out RG from 81.7 to 85.9 but reduces OOD GQA from 28.7 to 16.3.Source: Table 5 · p. 10, §3.9 · p. 10

3.10. Generalization to external tool-calling

• 55.8 BFCL accuracy on Qwen3-8B, up from 43.3, demonstrates transfer to externally specified APIs despite unseen schemas, multi-turn traces, and judges.Qwen3-4B also improves from 45.1 to 48.6 on BFCL v4’s no-web subset.Source: Table 6 · p. 10, §3.10 · p. 10

3.11. Ablation

• 68.8 RG (Unseen) for single-model Tool Create versus 58.9 for Decoupled Build/Use shows joint training matters, while only full SMITH achieves the best aggregate score.The ablation attributes robustness to separating schema, code, and outcome quality through distinct reward components.Source: Table 7 · p. 11, §3.11 · p. 11

Scale, judge dependence, and base priors

• SMITH’s gains may not persist, saturate, or invert at 70B+ scale, while removing the external judge improves held-out RG but degrades OOD GQA.All trained policies are at most 8B parameters, whereas the quality judge has 30B activated parameters; the Self-Judge ablation is only a partial probe of judge-size dependence.Source: §4 · p. 11
• SMITH does not formally certify that adversarial prompts cannot induce unsafe tools and reports no human evaluation of schema readability or developer-facing reusability.Generated Python executes at every step inside a sandbox, but safety certification and human assessment remain absent.Source: §4 · p. 11

Open Questions

• SMITH represents each tool as one Python function plus a JSON schema, so generating richer multi-file or server-backed artifacts would require extending its output representation and validation pipeline.Potential targets include SKILL.md packages with scripts and resources and MCP servers exposing tools, resources, prompts, and instructions; iterative or multi-turn generation is left for future work.Source: §4 · pp. 11–12
• SMITH never produced parallel tool calls or multiple tools in one turn, likely because of base-model single-tool tendencies and reward design, despite parallel calls being common in deep research.Extending SMITH to multi-tool generation and parallel execution is left to future work.Source: §4 · p. 12

Conclusion

• SMITH jointly trains a single language model to create and use reusable tools, optimizing the policy directly on its own execution outcomes.This closes the feedback loop between tool writer and tool user.Source: §5 · p. 12
• 79.8 held-out RG accuracy makes the trained 4B model best among evaluated methods on held-out Reasoning-Gym tasks, while it also leads TabMWP-Hard.These results come from training on 13 Reasoning-Gym tasks.Source: §5 · p. 12
• Tools from the 4B model transfer to a 350 M student and match tools from a model an order of magnitude larger, while the recipe lifts Qwen3-8B and Granite-3.3-8B unchanged.The models were not exposed to the relevant training data for these transfer results, as specified in the supplied conclusion passage.Source: §5 · p. 12

A. Related works

• Existing systems create tools through inference-time prompting, whereas SMITH jointly trains creation and use across 13 task categories, validates tools on up to 16 held-out questions, and adds LLM-judge rewards.The related-work landscape includes LATM [2], CRAFT [35], Trove [31], KTCE, and SAGE [28], but these do not match SMITH’s joint objective and validation setup.Source: A. Related works · p. 16

B.1. System Overview

• SMITH jointly trains build and use tasks: the model generates Python code and an OpenAI-compatible schema, receives structural, held-out evaluation, and judge signals, then invokes pooled tools under rule-based answer matching.Build rewards separately assess schema alignment, tool performance, and code quality, while use tasks require correct invocation of a supplied schema.Source: §B.1 · pp. 16–17

B.2. Observed Failure: build/env_reward Collapse

• At step 6, 46/238 build states failed schema checks versus 5/197 at step 5, driving the collapse while evaluated items still averaged 5.26/8 versus 5.44/8.The evaluator produced no connection errors or exceptions; the reward drop was almost entirely caused by forced zeros for step1 failures.Source: §B.2 · p. 17

B.3.1. Naming Drift

• At step 6, naming drift caused the schema to reference a function absent from the generated code, although the code contained several plausible function names.Figure 2 gives a representative case in which the schema names find_lcm without a matching top-level callable.Source: §B.3.1 · p. 18, Figure 2 · p. 18

B.3.2. Reward Signal Misalignment

• The verifier terminated schema failures before judge scoring, so the judge rewarded multi-function outputs that looked clear even when their schema function was absent, reinforcing the failure through the next gradient update.A verifier-enforced hard failure must receive zero judge credit; otherwise partial judge credit creates a gradient toward verifier-failing outputs.Source: §B.3.2 · pp. 18–19, Table 9 · p. 18

B.3.3. Secondary Factor: LoRA-Synced Evaluator

• The LoRA-synced evaluator was a secondary factor: mean per-item accuracy fell only from 5.44 to 5.26 out of 8, while the step1-failure increase dominated the collapse.The post-sync accuracy dip is expected because tool-use ability can lag behind tool-building improvements early in reinforcement learning.Source: §B.3.3 · p. 19

B.4.1. Lesson 1: Hard Failures Must Be Hard Everywhere; B.4.2. Lesson 2: Align the Judge Prompt to the Verifier’s Hard Constraints

• Hard verifier failures should receive zero or negative judge reward, and every verifier-enforced structural constraint should be stated explicitly in the judge prompt with scored counterexamples.The revised prompt required the schema name to appear verbatim as a top-level def and penalized thin wrappers or indirect multi-function implementations.Source: §B.4.1 · p. 19, §B.4.2 · pp. 19–20

B.4.3. Lesson 3: The Reward Gap Between Verifier and Judge Is a Gradient Leak; B.4. Lessons for Reward and Judge Design

• With additive rewards, a verifier-failed output can earn positive total reward—for example 0.2 when r_j=0.4 and w_j=0.5—creating a gradient leak toward structural failures.The recommended fix is to set r_j=0 whenever the verifier assigns zero, or use judge scores only as a tiebreaker among verifier-passing outputs.Source: §B.4.3 · pp. 20–21

B.4.4. Lesson 4: Log Structural Sub-Metrics as First-Class Signals

• fn_mismatch fell from 23.4% at step 1 to 0.0% at step 5 while param_mismatch stayed at 1–3%, yet structural failure appeared at step 6.Aggregate reward looked healthy and would have missed the impending failure, demonstrating why structural sub-metrics must be first-class monitoring signals.Source: §B.4.4 · p. 21
• A step1_fail alert above 10% would have caught the failure at step 6 before eval_reward collapsed at step 7.Recommended monitoring includes step1_fail, fn_mismatch, and param_mismatch as independently tracked structural constraint-violation rates.Source: §B.4.4 · p. 21

B.4.5. Lesson 5: Emergent Style Transfer Can Corrupt Structural Constraints

• At steps 3–5, helper-function style correlated with high 𝑟eval, but at step 6 it overrode schema naming and produced fn_mismatch.The failure was reward hacking: a style associated with earlier correctness generalized beyond the structural constraint and was visible only in sub-metric trajectories.Source: §B.4.5 · p. 22

B.5. Summary of Design Principles

• Table 10 summarizes five actionable design principles for combining a structural verifier with an LLM-as-judge reward signal in RL code-generation training.The surrounding lessons emphasize independent structural signals, threshold alerts, and hard-coded verifier constraints rather than soft proxies alone.Source: Table 10 · p. 22, §B.5 · p. 22, §B.4.5 · p. 22

C. LLM-as-Judge Prompt (v5.0)

• The judge evaluates generated Python tools and OpenAI-compatible schemas using structural checks, task coverage, schema–code comparison, API design, and scored quality dimensions.Its rubric includes code_correctness, code_clarity, schema_quality, schema_code_alignment, and overall_quality, with overall_quality capped at code_correctness + 1.Source: C. LLM-as-Judge Prompt (v5.0) · pp. 23, 27

D.1. Build-Task Prompt

• Build prompts combine fixed tool-writing instructions, N=4 easy-band question–answer examples, and hidden metadata stripped before generation.The instructions require atomic typed parameters, Verb-Noun names, exact schema–function alignment, standard libraries, explicit edge-case handling, and exactly one Python and one JSON block.Source: §D.1 · p. 28

D.2. Use-Task Prompt

• Use prompts present only an OpenAI-compatible tool schema, testing whether its interface is clear enough to drive correct invocation without exposing the Python implementation.All appendix prompt templates remain fixed throughout SMITH training, with no prompt engineering between runs.Source: D. Build and Use Task Prompt Templates · p. 28, §D.2 · p. 29

D.3. Evaluator Prompt for 𝑟eval

• Evaluator reward counts only answers produced through a successful tool call, using schema-only access and LLM equivalence judging with exact matching as a fallback.The evaluator never sees the Python implementation, preventing text-only reasoning from bypassing the tool-use objective.Source: §D.3 · p. 29

E. Tool Pool Design

• The Tool Pool caches valid generated tools so later use rollouts can invoke them directly without repeating tool construction within or across batches.The pool decouples creation and use while retaining category-specific tools and evaluation scores.Source: E. Tool Pool Design · p. 30

E.1. Pool Structure

• The pool is a thread-safe category-keyed dictionary storing Python functions, OpenAI-compatible schemas, and admission-time evaluation scores.A re-entrant lock protects concurrent rollout-worker access.Source: §E.1 · p. 30

E.2. Admission Criterion

• A generated tool enters the pool only when 𝑟eval > 0, meaning it answers at least one of 16 held-out questions correctly; zero-score tools are discarded.Format failures and execution errors are therefore never cached.Source: §E.2 · p. 30

E.3. Capacity Cap and Eviction Policy

• Each category stores at most 20 tools, evicting the lowest-scoring entry and using oldest insertion as the tie-breaker.This quality-first policy raises the pool’s minimum quality over training, creating an implicit difficulty curriculum.Source: §E.3 · p. 30

D.2. Use-Task Prompt

• Each use-task rollout receives one domain schema and two distractors, forcing schema identification and valid argument construction rather than defaulting to the first tool.The model interacts through up to five tool-call turns, and the final answer must use a boxed delimiter.Source: §D.2 · p. 29, §E.4 · p. 31

E.4. Retrieval and Distractor Selection

• For each category, retrieval selects one recent domain tool and up to two randomly sampled distractors from other nonempty categories before injecting all schemas into the prompt.If no domain tool exists, the rollout performs a fresh build pass first.Source: §E.4 · p. 31

E.5. Initialisation

• The pool starts empty, while balanced build and use batches populate uncovered categories; once a category has one valid tool, retrieval skips building and saves approximately one forward pass.This initialization path accelerates early coverage and reduces repeated construction for established categories.Source: §E.5 · p. 31

F. Full Easy-to-Hard Difficulty Split for All 13 Training Tasks

• Band 2 supplies induction examples while each evaluation set uses the hardest available band, preventing memorization from solving the evaluation instances.The gap is defined by progressively harder task-generator parameters, so successful tools must encode the underlying algorithm.Source: F. Full Easy-to-Hard Difficulty Split… · p. 31
• 500 Band 2 instances train each task, with an additional 500 Band 3 instances for tasks needing a larger difficulty bridge.At rollout time, r_eval uses K=16 hardest-band instances, while final accuracy is measured over 70 instances per difficulty level.Source: F. Full Easy-to-Hard Difficulty Split… · p. 31
• The 13 categories cover arithmetic, algorithms, algebra, games, and logic, exposing the policy to diverse algorithm types rather than one reasoning pattern.The categories include bitwise arithmetic, cryptarithmetic, bit counting, LCM, GCD, base conversion, isomorphic strings, polynomial tasks, countdown, Tower of Hanoi, knights and knaves, and Caesar cipher.Source: F. Full Easy-to-Hard Difficulty Split… · p. 32

G. OOD Inference Protocol for TabMWP-Hard and GQA

• OOD evaluation mirrors the Reasoning-Gym loop as closely as possible, isolating domain shift from differences in evaluation procedure.The protocol is specified for TabMWP-Hard and GQA.Source: G. OOD Inference Protocol for… · p. 32

G.1. Single Build-Then-Use Loop

• Each benchmark uses one build pass followed by applying the resulting tool to every test instance, with the model generating Python code and an OpenAI-compatible JSON schema.The build pass uses reference question-answer pairs from the benchmark’s training or validation split.Source: §G.1 · p. 32
• A failed build discards the tool and assigns zero to the entire run, while each successful use permits up to T=5 calls and scores one only when the returned answer matches ground truth.Failures include syntax errors, schema-code mismatches, and execution errors; the induced tool remains unchanged across test instances.Source: §G.1 · p. 32

G.2. In-Context Examples

• TabMWP-Hard uses N=4 training triples with pipe-delimited tables, whereas GQA uses N=10 question-answer pairs without images to infer visual-primitive composition.The same TabMWP-Hard examples are reused across evaluation runs; GQA uses more examples because its visual question types are more diverse.Source: §G.2 · pp. 32–33

G.3. Protocol Differences from the RG Training Loop

• OOD evaluation uses same-distribution in-context examples and test instances, omits the training Tool Pool, and reports only final test-split accuracy.These changes make OOD performance reflect generalization of tool-writing strategy rather than exploitation of training-distribution shortcuts.Source: §G.3 · p. 33

H. GQA Visual Tool Setup

• GQA evaluation supplies three pre-implemented visual primitives, letting text-only SMITH compose reusable tools without implementing low-level vision code.The primitives are served through a local API, while the model performs the build task over these functions.Source: H. GQA Visual Tool Setup · p. 33, §H.1 · p. 33

H.1. Visual Primitive Functions

• The visual API detects named objects with OWL-ViT, answers image questions with BLIP-VQA, and crops the first detected box with 1.5× padding.All three functions are served by a configurable FastAPI server at localhost:8000.Source: §H.1 · pp. 33–34

H.2. Image Encoding

• Each GQA image is base-64 JPEG encoded into the global IMAGE variable, enabling generated tools to use a fixed solve(question: str) -> str interface.The generated code reads IMAGE rather than accepting an image path or URL.Source: §H.2 · p. 34

H.3. Build-Task Prompt for GQA

• The GQA build prompt prepends primitive signatures and docstrings, requiring composition of those functions instead of reimplementing vision logic.The injected constraint requires tools to accept only question: str and return a plain string while calling the provided primitives directly.Source: §H.3 · p. 34
• The build task provides N=10 uniformly sampled GQA training examples as question–answer pairs without images, forcing reasoning about image content through the available primitives.Images are absent from the prompt itself, so the model must use the primitive interface to access visual information.Source: §H.3 · p. 34

H.4. Tool Selection and Evaluation

• Ten candidate tools are generated from ten GQA training examples, evaluated on 100 disjoint validation questions, and the highest-accuracy candidate is tested on GQA.Failed builds receive zero validation accuracy; if all candidates fail, an empty-response fallback scores zero on every test question.Source: §H.4 · pp. 34–35

H.5. Answer Verification

• GQA uses lowercased, whitespace-stripped exact string match over 2,516 test questions, without an LLM equivalence judge.The short free-form answer format makes rule-based verification sufficient while avoiding judge overhead.Source: §H.5 · p. 35

I. ReTool Evaluation Setup

• ReTool [6] is evaluated as a distillation baseline to isolate schema-grounded tool representation: both methods use execution feedback, but only SMITH creates reusable callable schemas.The setup documents the checkpoint, training configuration, and evaluation protocol for ReTool.Source: §I · p. 35

I.1. Checkpoint and Training Configuration

• ReTool fine-tunes Qwen3-4B-Instruct on the official ReTool-SFT dataset, which contains code-execution trajectories distilled from Qwen-32B.The training uses QLoRA, with the dataset hosted as JoeYing/ReTool-SFT on HuggingFace.Source: §I.1 · p. 35
• ReTool matches SMITH’s LoRA configuration of r=64 and α=128, attributing performance differences to the training objective rather than adapter capacity.This matching configuration is intended to control for adapter capacity in the comparison.Source: §I.1 · p. 35

I.2. Evaluation Protocol on Reasoning-Gym and OOD Benchmarks

• On Reasoning-Gym and TabMWP-Hard, ReTool receives each question directly, generates fresh code without a tool pool or build pass, and may execute up to 10 rounds.Episodes terminate when the model emits an answer block or exhausts the turn budget; the final answer is extracted from the boxed delimiters.Source: §I.2 · p. 35
• For GQA, ReTool receives descriptions of locate_objects, visual_qa, and crop_region, while their implementations are prepended to every execution block for direct visual calls.These additions let ReTool incorporate vision through direct function calls rather than a reusable composed schema.Source: §I.2 · p. 36

I.3. Key Distinction from SMITH

• SMITH stores callable JSON schemas in a Tool Pool for reuse across questions, whereas ReTool discards ephemeral code after each turn.This distinction removes the need to rerun tool construction for every question.Source: §I.3 · p. 36
• SMITH trains end-to-end from verifiable rewards without a strong oracle, whereas ReTool uses behavioral cloning from a 32B oracle.The training signals therefore differ in whether supervision comes from explicit reward or oracle demonstrations.Source: §I.3 · p. 36
• 92.0 versus 86.6 on RG (Seen) favors ReTool, while SMITH gains the structural edge on RG Unseen, GQA, and TabMWP-Hard through reusable schemas and RL generalization.ReTool’s seen-task advantage reflects distillation from a 32B oracle, whereas SMITH’s advantage emerges on held-out transfer.Source: §I.3 · p. 36

J. LATM (distill GPT-4.1) Baseline Setup

• The LATM baseline distills GPT-4.1 tool-writing trajectories into Qwen3-4B across all 13 training task categories.GPT-4.1 generates trajectories using the SMITH build-task instructions and five easy-band question-answer examples per category.Source: J. LATM (distill GPT-4.1) Baseline Setup · p. 36, §J.1 · p. 36

J.1. Trajectory Collection

• Trajectory collection retains both oracle tool-making outputs and successful tool-use conversations, collecting up to 1,000 balanced conversations per category.Tool use is retained only when calls execute without errors and answers match ground truth; generation uses up to 10 call rounds.Source: §J.1 · pp. 36–37
• Distractor augmentation adds one distractor to 50% of retained conversations and a second to 90%, producing tool-making and tool-use data for all 13 categories.The augmentation follows SMITH’s pool-retrieval logic to test tool selection under comparable conditions.Source: §J.1 · p. 37

J.2. Fine-Tuning Configuration

• LATM uses full-parameter fine-tuning rather than LoRA, testing reward-versus-cloning training signals while giving the distilled model greater capacity to memorize oracle style.SMITH uses LoRA with r=64 and α=128, whereas LATM updates all model weights.Source: §J.2 · p. 37

J.3. Evaluation Protocol

• At test time, LATM autonomously builds and uses tools from N=4 in-context examples without an oracle, with results reported on RG (Seen), RG (Unseen), TabMWP-Hard, and GQA.Evaluation follows the standard build-then-use inference protocol, with results reported in Tables 1 and 4.Source: §J.3 · p. 37

J.4. Comparison to SMITH

• 79.8 vs. 66.4 on RG Unseen and 42.6 vs. 56.0 on GQA show SMITH’s transfer advantage over LATM, despite LATM reaching 83.3 on RG Seen.LATM imitates GPT-4.1 outputs, whereas SMITH updates from verifiable held-out execution rewards without oracle demonstrations; SMITH’s GQA score uses no visual oracle data.Source: §J.4 · pp. 37–38

K. Agentic Loop Comparison: KTCE, CRAFT, TroVE, and SMITH

• KTCE creates and evolves a verified offline toolset before testing, CRAFT builds a diversity-sampled offline library, and TroVE grows its library online as problems arrive.KTCE uses evaluate/delete/modify/add/rollback iterations; CRAFT generates and abstracts executable tools; TroVE runs import, create, and skip modes with frequency-based forgetting.Source: §K.2 · p. 38, §K.3 · pp. 38–39, §K.4 · p. 39

Semantic embeddings

• The baselines rely on substantial construction-time prompting and embeddings, while SMITH replaces those procedures with RL fine-tuning and amortized rollout training.KTCE uses BGE-M3 clustering and retrieval, CRAFT uses SimCSE diversity sampling and four retrieval views, and SMITH requires RL fine-tuning rather than test-time semantic lookup.Source: §K.2 · p. 38, §K.3 · p. 39, Semantic embeddings · p. 42

Tool reuse across problems

• TroVE’s online reuse makes results order-dependent: later problems benefit from earlier tools, and shuffling the problem sequence changes outcomes.TroVE prunes low-frequency tools every 500 problems and regenerates problems whose winning tools were removed.Source: §K.4 · p. 39, Tool reuse across problems · p. 41

K.5. SMITH: RL-Trained Tool Creation Coupled with Tool Use

• SMITH trains a 4B policy with RL to jointly create and use tools, generating a Python function and matching JSON schema while optimizing execution, code-quality, and format rewards.Build-task correctness requires a successful tool invocation, while use tasks invoke shared-pool tools against held-out questions; only execution-verified tools enter the pool.Source: §K.5 · pp. 39–40, Optimization / refinement · p. 41

Agent loop at inference

• At inference, SMITH uses approximately 3 LLM calls: one-pass build generation, tool invocation and execution, followed by answer synthesis.Source: §K.5 · p. 40, Agent loop at inference · p. 41

K.7. Key Conceptual Distinctions

• SMITH eliminates test-time retrieval and presents an OpenAI function-calling schema, unlike KTCE’s subfield lookup, CRAFT’s multi-view retrieval, and TroVE’s full-library prompting.SMITH generates the needed tool directly in one pass, then invokes it externally before synthesising the answer; baseline systems expose raw code or retrieved libraries to the model.Source: §K.2 · p. 38, §K.3 · p. 39, §K.4 · p. 39

Optimization / refinement

• RL is SMITH’s optimization loop, whereas KTCE uses explicit loss-guided evolution, TroVE uses frequency-based forgetting, and CRAFT performs no post-creation optimization.The comparison table summarizes these dimensions across KTCE, CRAFT, TroVE, and SMITH.Source: Table 15 · p. 40, Optimization / refinement · p. 41, §K.7 · p. 42

L.1. KTCE: Three Compounding Failure Modes

• KTCE’s dispatcher solves bitwise_arithmetic, count_bits, tower_of_hanoi, and isomorphic_string at 100% without LLM calls, masking tool-generation failure.The evolutionary tools for these categories were never invoked.Source: §L.1 · p. 43
• KTCE’s incomplete or absent dispatcher yields cryptarithm at 5.7%, knights_knaves at 4.8%, polynomial_equations at 1.1%, and GQA at 0%.GQA reaches 0% because the dispatcher has no visual-question-answering implementation.Source: §L.1 · p. 43
• For approximately half the tasks, KTCE generates stub tools whose bodies simply return "", so successful execution still produces unusable answers.This makes performance depend on extracting an answer from the raw model response.Source: §L.1 · p. 43
• Stub-tool tasks retain high textual-reasoning accuracy—syllogism 96.7% and gcd 96.2%—but computation-heavy tasks collapse: ab 0%, group_anagrams 0%, and base_conversion 0%.KTCE therefore effectively reduces to a plain chain-of-thought baseline when tools fail.Source: §L.1 · p. 43
• Intermittent LLM endpoint failures leave ab with real responses for only 64 of 210 entries and can overwrite aggregate token counts with zeros.Token usage is therefore read from individual inference files while entries with completion_tokens = 0 are skipped.Source: §L.1 · p. 43

L.2. TroVE: Missing Token-Usage Records

• TroVE stores token usage only when the backend records it; across Phase 2’s 20 tasks, 12 have no stored token_usage.Phase 1 grows the library, while Phase 2 scores with the frozen library.Source: §L.2 · p. 43
• Reconstructing TroVE prompts with the initial three-function toolbox produces only ∼179 characters (∼45 tokens) instead of the larger learned-library prompts, underestimating usage as 691 tokens.Phase 2 prompts include the substantially larger library learned during Phase 1.Source: §L.2 · p. 44
• 829 tokens is TroVE’s corrected macro-average over the 8 of 20 tasks with recorded usage, and it is a lower bound because excluded tasks likely have larger learned libraries.The true average therefore likely exceeds 829 tokens per question.Source: §L.2 · p. 44

M. TabMWP-Hard: Dataset Construction and Augmentation; M.1. Table Type Coverage

• TabMWP-Hard addresses the weakness of 96.8% Standard CoT performance on original TabMWP by targeting lookup shortcuts with structurally suitable table types.Financial Ledger, Price List, and Two-Way tables are selected because they support arbitrarily large distractor pools and adversarial near-miss construction, unlike Stem-and-Leaf and Function Tables.Source: M. TabMWP-Hard: Dataset Construction… · p. 44, §M.1 · p. 44

M.2. Augmentation Pipeline

• Each selected entry undergoes distractor injection, near-miss insertion, irrelevant-column augmentation, and a two-step validity gate before collision checking.Up to 2,000 rows are added, near-miss keys are placed beside targets, and failed original-table or augmented-table answer checks cause entries to be discarded.Source: §M.2 · p. 44

M.3. Worked Example A: Two-Way Table

• The Two-Way example transforms a trivial five-row lookup into a 308-row, seven-column table with buried targets, adjacent near-miss names, and four irrelevant columns.The resulting task requires locating the correct rows, reading the relevant values, and ignoring plausible neighboring distractors.Source: §M.3 · pp. 44–45

M.4. Worked Example B: Price List Table (Missing Header)

• The Price List example adds compound-name distractors and semantic near-misses, then infers a header and expands the table to 316 rows and six columns.This simultaneously tests header construction and resistance to clusters of plausible but incorrect shell names surrounding the target rows.Source: §M.4 · pp. 45–46

M.5. LLM Column Proposal Mechanism

• gpt-oss-120b proposes up to four new columns from the original table, supplies Python code producing exactly n values, and uses deduplication to prevent duplicate column names.For headerless Price Lists, it also proposes names for the existing columns before constructing the injected header.Source: §M.5 · p. 46

M.6. Collision Avoidance

• Collision avoidance protects ground-truth integrity by filtering duplicate keys, preserving ledger dates, validating answer-value near-misses, and discarding entries without surviving adversarial rows.The process combines deterministic key checks, rule-based duplicate removal, and LLM review of value correctness, formatting, and semantic-domain consistency.Source: §M.6 · pp. 46–47

N. Compute and Training Cost

• SMITH training runs required 36–72 hours per model-size and tool-pool configuration on a single NVIDIA RTX 6000 Pro GPU, without multi-node clusters.The reported range varies with rollout length, pooled-tool count, and policy-backbone size.Source: N. Compute and Training Cost · p. 47

O. TroVE Baseline: A Controlled Prompt-Tuning Ablation; O.1. Versions compared •

• The TroVE ablation tests prompt sensitivity with Qwen3-4B-Instruct-2507 under its frozen-toolbox protocol, comparing the default prompt with bundled, punctuation-only, and worked-example-only revisions.The bundled revision combines three changes, while v3 and v4 isolate punctuation and the second worked example; the empty-toolbox fallback is a harness-level no-op.Source: O. TroVE Baseline: A Controlled… · p. 47, §O.1 · p. 47

Prompt ablations

• On regressing tasks, v3 and v4 significantly reduce caesar_cipher accuracy to 49.5% and 51.5% from v1’s 59.2%, while v3 reduces polynomial_multiplication to 45.8% from 52.7%.For polynomial_equations, only v3 significantly regresses, from 37.9% to 31.4%; ceiling and near-ceiling tasks remain largely unchanged.Source: §O.2 · p. 48
• The worked-example change leaves performance closer to v1 than the punctuation instruction, implicating ASCII punctuation as the larger source of regression on polynomial tasks.On caesar_cipher, the two changes contribute comparably; v4 is statistically indistinguishable from v1 on polynomial_equations.Source: §O.2 · p. 48

Grading caveats

• Regrading removes the apparent bitwise_arithmetic prompt effect, yielding near-ceiling accuracies of 99.3%, 99.3%, 97.5%, and 100.0% for v1–v4, respectively.Symbolic regrading raises polynomial_multiplication accuracy for every version but preserves the significant v1–v2, v1–v3, and v1–v4 gaps.Source: §O.2 · p. 48

Scope and exclusions

• On GSM8K (N=1319), v1 reaches 91.1% versus 90.2% for v2 and 90.8% for v3, while tower_of_hanoi and cryptarithm are excluded because duplicated prompts invalidate paired significance tests.Neither tested version significantly outperforms v1 in the larger-scale spot check.Source: §O.2 · p. 48

Overall results

• No configuration significantly outperforms v1 across the eight matched reasoning tasks, so non-ablation TroVE results use the original prompt.Table 17 evaluates strict exact-match accuracy with paired two-sided exact McNemar tests; v1 is the strongest of the four tested versions.Source: §O.2 · p. 48, Table 17 · p. 51