PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models

Jonggeun Lee, Woojung Song, Jongwook Han, Haesung Pyun, Yohan Jo

arXiv:2510.07248v3cs.CL

TL;DR

Small language models often fail tool selection and parameter identification when schema names conflict with naming patterns learned during pretraining. PA-Tool adapts tool schemas by generating candidate names and selecting those with the highest peakedness, a pretraining-familiarity signal. It improves tool-use performance by up to 17% and reduces schema-misalignment errors by 80%, while remaining training-free; its reliance on peakedness and English-language schemas limits supported generality.

  • Problem

    Small language models struggle with tool selection and parameter identification, often hallucinating plausible but nonexistent components when provided schemas use unfamiliar naming conventions.

  • Method

    PA-Tool generates multiple names for each schema component, scores candidate clustering with peakedness, and selects the highest-scoring pretrained-aligned name without retraining the model.

  • Results

    PA-Tool improves tool-use performance by up to 17% across MetaTool and RoTBench and reduces schema-misalignment errors by 80%.

  • Takeaways & Limitations

    A simple schema-level name mapping can improve small-model tool use while preserving computational efficiency and avoiding model training or fine-tuning.

  • Takeaways & Limitations

    The method assumes peakedness reliably indicates pretraining familiarity and is evaluated on English-language schemas, so effectiveness may vary across training distributions and languages.

Abstract

from arXiv · show

Small language models (SLMs) enable scalable tool-augmented multi-agent systems where multiple SLMs handle subtasks orchestrated by a powerful coordinator. However, they struggle with tool-use tasks, particularly in selecting appropriate tools and identifying correct parameters. A common failure mode is \textit{schema misalignment}: models hallucinate plausible tool names that are absent from the provided tool schema, due to different naming conventions internalized during pretraining. Rather than training models to adapt to unfamiliar schemas, we propose adapting schemas to align with models' pretrained knowledge. We introduce \textbf{PA-Tool} (Pretraining-Aligned Tool Schema Generation), a training-free method that leverages peakedness, a signal used in contamination detection that indicates pretraining familiarity, to rename tool components. By generating multiple candidates and selecting the candidate with the highest peakedness, PA-Tool identifies pretraining-aligned naming patterns. Experiments on MetaTool and RoTBench show improvements of up to 17\%, with schema misalignment errors reduced by 80\%. PA-Tool enables small models to substantially improve tool-use accuracy without retraining, showing that schema-level interventions can unlock the tool-use potential of resource-efficient models. Our code is available at https://github.com/holi-lab/PA-Tool.

1 Introduction

Small language models struggle with tool selection and parameter identification because unfamiliar schemas trigger plausible but nonexistent tool components. PA-Tool addresses this mismatch by adapting schema names to pretrained knowledge, improving tool-use performance without retraining.

  • Schema misalignment causes SLMs to hallucinate plausible tool components absent from the provided schema, reflecting naming conventions internalized during pretraining.The hypothesized remedy is to align component names with those internalized patterns.
  • PA-Tool generates candidate names, scores their peakedness as a familiarity signal, and maps original tool components to the highest-scoring alternatives.The method is training-free and adapts schemas rather than modifying models.
  • On RoTBench, tool-selection gains reach 10% in single-turn and 6% in multi-turn settings, with consistent improvements in parameter identification.These results indicate that alignment benefits persist across extended contexts.
  • Schema-misalignment errors decrease by 80.0% with PA-Tool, while other error types also decline by 18.8–24.0%.The analysis identifies schema misalignment as the predominant SLM failure mode addressed by the method.
  • PA-Tool improves end-to-end task completion on API-Bank and τ-Bench and can complement fine-tuning and other training-free methods.The approach requires only a one-time schema mapping and no model retraining or modification.
  • Up to 17% improvement across SLMs on MetaTool and RoTBench extends from tool selection to parameter identification in single-turn and multi-turn settings.The gains cover diverse tool-use scenarios rather than a single task.

2 Related Work

Related work studies tool-use evaluation, agentic systems, and contamination detection, but largely leaves schema-level mismatch unaddressed. PA-Tool repurposes peakedness from black-box memorization detection to construct pretrained-aligned tool schemas.

  • Small language models in multi-agent systems can cause cascading failures through incorrect tool selection and parameter identification, including plausible tool names absent from actual schemas.Existing work primarily improves agent architectures or coordination strategies rather than the schema mismatch itself.
  • Tool-use research spans fine-grained benchmarks for tool selection and parameter identification as well as end-to-end multi-step evaluations.These efforts establish evaluation settings ranging from individual decisions to multi-turn task completion.
  • Peakedness-based contamination detection identifies memorized patterns from sampled candidates without requiring token probabilities or white-box model access.PA-Tool adapts this mechanism from detection into constructive schema optimization.

3 PA-Tool: Pretraining-Aligned Tool Schema Generation

PA-Tool generates pretraining-aligned names for tool-schema components by sampling candidates, measuring their peakedness, and selecting the most familiar naming patterns. The process is applied across the schema hierarchy to produce a renamed schema.

  • 3.1 Framework Overview: PA-Tool samples multiple names for each schema component from the model, computes peakedness, and selects the highest-scoring candidate as its pretraining-aligned name.The method constructs a mapping from original component names to aligned alternatives by applying the process to each component in the schema hierarchy.
  • 3.1 Framework Overview: The framework comprises candidate generation, peakedness computation, and schema selection stages, summarized in Figure 2.Candidate names are sampled beyond the single greedy decoding path, while greedy decoding provides a reference name for tie-breaking.
  • 3.3 Stage 2: Peakedness Computation: Peakedness counts nearby candidate names under a length-adaptive edit-distance threshold, identifying naming patterns the model generates consistently.The threshold scales with the maximum candidate length through α, allowing proportionally more variation for longer names.
  • 3.4 Stage 3: Schema Selection: The representative name is the candidate with maximum peakedness, with minimum edit distance to the reference name resolving ties.This selection favors patterns that are both consistently generated and, when tied, closest to the reference name.
  • 3.4 Stage 3: Schema Selection: Iterating selection across all tools and parameters produces the final pretraining-aligned schema.The approach treats high-peakedness regions as memorized naming conventions that are natural according to the model’s learned knowledge.

4 Experimental Setup

The experiments evaluate PA-Tool primarily on small open-source language models across MetaTool and RoTBench tool-use settings, using multiple schema-generation configurations and closed-source reference models. Accuracy is measured by exact agreement with benchmark ground-truth labels.

  • Benchmarks: MetaTool contains 4,287 test cases with 199 tools and evaluates Similar, Scenario, Reliability, and additional tool-selection subtasks.The supplied passage explicitly describes the first three subtasks and begins the fourth without providing its full description.
  • Models and Configurations: The study evaluates four open-source SLMs—Qwen2.5-3B/7B, Llama3.1-8B, and Llama3.2-3B—across five schema configurations.Configurations include the original Base schema, Greedy decoding, and MostFreq selection; the supplied passage truncates the remaining configuration names.
  • Baselines: The experiments also compare SLM performance with GPT-4.1-mini, Gemini-2.5-Flash, and Claude-Sonnet-4.5 as closed-source reference models.These models contextualize the performance of the evaluated small language models.
  • Implementation and Evaluation: PA-Tool schema generation uses 32 candidates at temperature 0.4 with α = 0.2, while benchmark inference uses temperature 0 for reproducibility.Accuracy is the primary metric, measuring the percentage of test cases with exact ground-truth matches.

5 Main Results

PA-Tool improves small-model tool use across MetaTool and RoTBench, with gains extending across tool selection, parameter identification, model families, and task settings.

  • MetaTool: Up to 17.0% gains on MetaTool Reliability and 9.6% on Multi-tool show substantial improvements over Base models.Reliability gains include Llama3.2-3B increasing from 43.6% to 60.6%, while Multi-tool gains include Llama3.1-8B increasing from 78.7% to 88.3%.
  • Comparison with Baselines: PA-Tool outperforms training-free alternatives on most tasks and is more consistent than single-candidate or frequency-based schema selection.Greedy can underperform Base because it explores only one candidate, whereas PA-Tool measures distributional concentration.
  • RoTBench: PA-Tool consistently improves RoTBench tool selection, with gains of 5.7–10.5% in single-turn and up to 6% in multi-turn settings.For Llama3.1-8B, single-turn accuracy rises from 58.1% to 68.6%.
  • RoTBench: RoTBench parameter-identification gains reach 4.3%, extending schema-alignment benefits beyond tool selection.Qwen2.5-3B multi-turn accuracy rises from 10.0% to 14.3%.
  • Comparison with Closed-source Models: On selected subtasks, PA-Tool narrows the gap with closed-source models and surpasses them in MetaTool Multi-tool.Llama3.1-8B reaches 88.3% versus Claude-Sonnet-4.5 at 85.1% on MetaTool Multi-tool.
  • Generalization to Diverse Models: PA-Tool improves diverse model families and scales, with larger gains for SLMs and complementary gains for reasoning models.Examples include 8.5% for Ministral-8B on RoTBench single-turn tool selection, 12.1% for GPT-4.1-mini on MetaTool Multi-tool, and up to 10.5% for Qwen3-1.7B parameter identification.

6 Analysis

PA-Tool analysis shows that schema alignment reduces naming-related tool-use errors, reflects pretraining familiarity, and complements training-based and training-free approaches across benchmarks.

  • Error Analysis: 80.0%: PA-Tool reduces Schema Misalignment errors, alongside reductions of 24.0% in Functional Confusion and 18.8% in Context Understanding.The analysis categorizes errors into schema misalignment, functional confusion, and context understanding, with PA-Tool reducing all three.
  • Validating Peakedness: Peakedness increases consistently with simulated pretraining exposure, reaching +25.8% for Llama3.2-3B across training epochs.The experiment computes loss on all tokens and measures peakedness at epochs 0, 10, 20, 30, 50, and 100.
  • Component Ablation: Joint tool-name and parameter-name alignment provides the most balanced RoTBench results, with each component benefiting its corresponding capability.Tool-name alignment improves tool selection, parameter-name alignment improves parameter identification, and the joint configuration is adopted as default.
  • Fine-tuning Compatibility: PA-Tool remains effective with supervised fine-tuning, and their combination achieves the best results on most subtasks, including 80.8% on Scenario.PA-Tool alone outperforms both SFT configurations on Reliability and Multi-tool, while SFT2 can degrade on RoTBench relative to SFT1.
  • Compatibility with Training-free Methods: PA-Tool complements retrieval, constrained generation, and description enhancement, with combined methods improving most subtasks and often producing the strongest results.Retrieval methods yield limited gains alone, constrained generation helps RoTBench but is mixed on MetaTool, and description enhancement addresses a separate schema-quality dimension.
  • Generalization and Human Evaluation: PA-Tool generalizes beyond selection benchmarks: it improves API-Bank accuracy and τ-Bench end-to-end completion, while human ratings favor its names over originals.On τ-Bench, schema misalignment cases for Llama3.1-8B decrease from 115 to 98; PA-Tool names receive significantly higher human ratings, with over 50% preference versus about 10% for originals.

7 Conclusion

The paper concludes that PA-Tool aligns tool schemas with pretrained knowledge through peakedness, improving small-model tool use without training. This schema-level intervention reduces misalignment and preserves computational efficiency for resource-constrained deployment.

  • Conclusion: PA-Tool uses peakedness as a pretraining familiarity signal to align tool schemas with small models’ pretrained knowledge.The method is training-free and operates through straightforward name mapping.
  • Conclusion: Up to 17%: PA-Tool improves tool-use performance while reducing schema misalignment errors by 80%.The conclusion presents schema adaptation as an effective strategy for enhancing small-language-model tool use without model training or fine-tuning.

Limitations

PA-Tool primarily targets SLMs, where schema misalignment is dominant, while its benefits diminish as model capacity reduces that problem. The method also assumes peakedness reflects pretraining familiarity, focuses on English schemas, and leaves descriptions unchanged.

  • Scope: PA-Tool primarily targets SLMs, where schema misalignment is a dominant failure mode, while gains diminish as model capacity increases.Even larger models can benefit when misalignment compounds, such as GPT-4.1-mini’s 12% gain on MetaTool Multi-tool.
  • Assumptions: The method assumes peakedness reliably indicates pretraining familiarity, but this relationship may vary across substantially different training distributions.This assumption was validated across the paper’s experiments.
  • Language scope: The evaluation focuses on English-language schemas, so character-level alignment may differ for non-Latin scripts or morphologically complex languages.
  • Design boundary: PA-Tool renames tool and parameter names but leaves descriptions unchanged because extending peakedness to descriptions risks semantic drift.Description enhancement remains compatible and produced complementary gains, while interaction-driven description refinement is left for future work.
  • Operational boundary: Name collisions remain below 6% across benchmarks, with lower rates on MetaTool than RoTBench because RoTBench has greater lexical overlap.MetaTool collision rates are 0–1.0%, while RoTBench rates are 4.2–5.8%.

A.4 Effect of Hyperparameters

PA-Tool’s performance is robust across its three hyperparameters on MetaTool, with the similarity threshold showing the clearest optimum. Candidate counts mainly need to scale with model size, while temperature has only marginal effects.

  • Number of Candidates: Smaller 3B models stabilize with 16–32 candidates, whereas 7–8B models require 32–64 before performance plateaus.Additional candidates beyond these ranges provide minimal gains.
  • Similarity Threshold: Performance peaks at α = 0.2 across all models, while α = 0.1 and α = 0.3 each reduce performance by 2–3%.
  • Sampling Temperature: Performance remains stable across t ∈[0.2, 1.0], varying within 1–2%, with only marginally better results at t = 0.4–0.6.

C Generalization to Diverse Models Details

PA-Tool generalizes across model families, architectures, scales, and training regimes, improving tool selection and parameter identification in most settings. Its gains are largest for smaller models and remain useful when schema misalignment compounds in larger or reasoning models.

  • SLMs: +8.5% improves Ministral-8B RoTBench single-turn tool selection from 66.7% to 75.2%, while GPT-4.1-nano gains +10.5% on parameter identification.Other SLM gains are more task-dependent, including Gemini’s +4.6% on MetaTool Reliability.
  • LLMs: +12.1% improves GPT-4.1-mini MetaTool Multi-tool from 72.2% to 84.3%, although average gains are smaller for larger models.The smaller gains are consistent with reduced schema misalignment at larger scales.
  • Reasoning Models: PA-Tool yields up to +10.5% on Qwen3-1.7B RoTBench single-turn parameter identification and +7.1% on multi-turn tool selection.Slight decreases occur on near-saturated subtasks, and reasoning and schema alignment address complementary failure modes.
  • Generalization: PA-Tool generalizes across diverse model families, architectures, and scales, improving performance in most additional-model settings.The evaluation covers 12 models across five families.
  • Fine-tuning: SFT + PA-Tool achieves the best results on most Qwen2.5 subtasks, including 89.5% Reliability and 77.8% Multi-tool for Qwen2.5-3B.The result generalizes complementary gains observed on Llama3.1-8B.

E Error Analysis Details

PA-Tool addresses schema-violating tool names, while retrieval, constrained decoding, and description enhancement address complementary aspects of tool use. Its benefits transfer across models and combine with other approaches, though self-generated schemas are usually strongest.

  • Error Taxonomy: PA-Tool reduces plausible but nonexistent tool-name errors by 80.0%, directly targeting schema misalignment rather than functional or contextual confusion.The error taxonomy distinguishes schema misalignment, selecting a similar existing tool, and selecting an unrelated tool.
  • Efficiency: One-time schema generation costs 8–16 seconds and can then be reused indefinitely without additional inference overhead.
  • Retrieval-based Correction: Retrieval correction rarely improves accuracy by more than 1% alone, but combining it with PA-Tool yields further gains on MetaTool and RoTBench.Retrieval corrects schema-violating outputs but cannot fix wrong selections among valid tools.
  • Constrained Generation: Constrained generation raises Qwen2.5-7B RoTBench single-turn tool selection from 49.5% to 78.1%, but can reduce MetaTool Reliability from 83.6% to 60.9%.Enum restrictions prevent invalid outputs but may bias models toward selecting a tool when none is appropriate.
  • Description Enhancement: PA-Tool outperforms EasyTool on several Llama tasks, while combining both achieves the best Reliability on three of four models and strongest overall parameter identification on RoTBench.The results indicate that naming alignment and description clarity address orthogonal aspects of schema quality.
  • Cross-model Transfer: Cross-model schemas improve over unaligned baselines, but self-generated schemas are optimal in most settings; larger same-family models provide a viable fallback.For example, Llama3.1-8B with Qwen2.5-7B schemas reaches 69.6% on Similar, +8.1% over Base.

I Robustness Across Noise Levels

PA-Tool robustness tests show that tool names are primary selection cues, with semantically meaningless or misaligned names causing substantial performance degradation. Renaming schemas with pretraining-aligned names improves tool selection across noise levels, multi-turn settings, and parameter identification.

  • Experimental Setup: Table 19 reports single-turn and multi-turn results across the Clean, Slight, Medium, and Heavy noise levels.These settings compare original schemas with character perturbations, semantically meaningless names, and shuffled names.
  • Noise-Level Effects: Slight name perturbations sharply reduce selection accuracy despite intact descriptions, showing that models rely heavily on tool names.Llama3.1-8B drops from 58.1% to 37.1%, while Llama3.2-3B drops from 56.2% to 34.8%.
  • Noise-Level Effects: Medium noise causes the largest performance drops, whereas Heavy noise partially recovers accuracy because shuffled names retain real-world naming patterns.For Llama3.1-8B, accuracy falls from 58.1% to 31.9% under Medium noise but reaches 48.6% under Heavy noise.
  • PA-Tool Results: +5.7 to +10.5 points: PA-Tool improves single-turn tool-selection accuracy over Clean schemas on the three stronger models.Examples include Llama3.1-8B increasing from 58.1% to 68.6%, Qwen2.5-7B from 49.5% to 55.2%, and Llama3.2-3B from 56.2% to 62.9%.
  • PA-Tool Results: PA-Tool improvements extend to multi-turn tool selection and parameter identification across the evaluated noise settings.The method surpasses the Clean baseline on nearly all settings while generating names from descriptions alone.
Loading 2510.07248v3…