PaperlayerView exact revision on arXiv

Source-linked AI summary

Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems

Qian Xiong, Yuekai Huang, Ziyou Jiang, Zhiyuan Chang, Yujia Zheng, Tianhao Li, Mingyang Li

arXiv:2507.15296v1cs.SEcs.AI

TL;DR

Parameter failures can undermine tool-agent execution, but their patterns and input-source effects have not been comprehensively characterized. The paper builds a failure taxonomy and studies perturbed inputs; it reports that hallucinated parameter names primarily reflect LLM limitations and that removing query parameters is associated with task deviation.

  • Problem

    Prior work often overlooks parameter handling and does not comprehensively examine how user queries, tool documents, and tool returns affect parameter filling.

    Source: §1 · p. 2

  • Method

    The study uses Grounded Theory to build a failure taxonomy from ToolLLaMa behavior and applies 15 perturbation methods to input sources to examine parameter behavior across four advanced LLMs.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Hallucinated parameter names primarily stem from inherent LLM limitations, while removing user-query parameters has over 50% failure rate for task deviation on almost all tested LLMs.

    Source: §6 · p. 6, §6.2 · p. 8

  • Takeaways & Limitations

    The five failure patterns distinguish how missing or redundant information, hallucinated names, specification mismatches, and task deviations can disrupt toolchains and task results.

    Source: §7 · p. 8

    The study mainly covers English, controlled single-turn conversations, and API-type tool invocations, leaving multilingual, real-time, multi-turn, and non-API settings outside its scope.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The emergence of the tool agent paradigm has broadened the capability boundaries of the Large Language Model (LLM), enabling it to complete more complex tasks. However, the effectiveness of this paradigm is limited due to the issue of parameter failure during its execution. To explore this phenomenon and propose corresponding suggestions, we first construct a parameter failure taxonomy in this paper. We derive five failure categories from the invocation chain of a mainstream tool agent. Then, we explore the correlation between three different input sources and failure categories by applying 15 input perturbation methods to the input. Experimental results show that parameter name hallucination failure primarily stems from inherent LLM limitations, while issues with input sources mainly cause other failure patterns. To improve the reliability and effectiveness of tool-agent interactions, we propose corresponding improvement suggestions, including standardizing tool return formats, improving error feedback mechanisms, and ensuring parameter consistency.

1 Introduction

Tool-agent execution depends on accurate parameter parsing, yet parameter errors can disrupt invocation and task completion. This study examines failure patterns and input-source effects to inform more reliable tool-agent design.

1 Introduction

  • Agents retrieve and plan tools, then parse existing inputs into parameters; incomplete queries, weak specifications, non-standard returns, and LLM limitations can cause errors.The examples include both incorrect parameter formats and values inconsistent with the user's intent.Source: §1 · p. 1
  • Parameter issues occur in about 44% of simple and 48% of complex user-query traces, limiting tool-agent usability and potentially disrupting task completion.Errors may confuse the invocation chain and reduce the quality of completed tasks.Source: §1 · p. 1
  • Prior work often focuses on particular tasks or overall completion, while few studies compare how user queries, tool documents, and tool returns affect parameter filling.This leaves the internal handling of parameters and the effects of different input sources less comprehensively examined.Source: §1 · p. 2
  • The study uses Grounded Theory with ToolLLaMa and applies 15 perturbation methods to three input sources, analyzing parameter behavior on four advanced LLMs.The paper's stated contributions include an API-invocation failure taxonomy and practical advice for improving tool-agent reliability.Source: §1 · p. 2

2 Related Work

Prior tool-use research evaluates capabilities such as tool use, selection, planning, and calling, but pays less attention to parameter handling during invocation. This paper focuses on that internal risk and the sources of parameter information.

2 Related Work

  • Tool-use evaluations cover model usage, understanding, selection, planning, tool calling, and real-world multimodal context inputs.Other work emphasizes robustness and security in tool use.Source: §2 · p. 2
  • Tool-agent research often emphasizes retrieval, selection, and final responses rather than risks during tool invocation.The paper identifies parameter filling as an influence on whether tools effectively support LLM problem-solving.Source: §2 · pp. 2–3
  • Existing research does not analyze parameter-information sources in depth, making it difficult for evaluations to reflect complex real-world input scenarios.The study responds by perturbing input sources and analyzing changes in parameter behavior.Source: §2 · p. 3

3 Failure Taxonomy Construction

The paper constructs a five-category taxonomy of parameter failures through qualitative analysis of ToolLLaMa behavior trajectories. The categories distinguish missing or extraneous information, invalid names or specifications, and values that deviate from the task.

3 Failure Taxonomy Construction

  • The taxonomy was developed from benchmark trajectories generated with ToolLLaMa, using open coding and constant comparison until no new patterns emerged.Three annotators refined consensus patterns until Cohen’s Kappa exceeded 0.9.Source: §3 · p. 3
  • Missing information occurs when required parameters are absent, producing imprecise tool results or preventing invocation.The tool receives less information than needed to solve the task.Source: §3 · p. 3
  • Redundant information adds parameters not mentioned by the user, potentially restricting returned results even when invocation succeeds.An example is limiting the number of results returned by a tool.Source: §3 · p. 3
  • Hallucination-name failure generates parameter names outside the tool’s recognized range, preventing normal invocation and response.The paper gives the “query” parameter for get_autocomplete as an example.Source: §3 · pp. 3–4
  • Task deviation occurs when parameter values conflict with task requirements, potentially yielding incorrect final solutions despite apparently successful invocation.The examples include mismatched regions, times, or ID attributes.Source: §3 · p. 4
  • Specification mismatch occurs when parameter types, ranges, or formats violate the tool documentation and prevent expected processing.Source: §3 · p. 4

4 Methodology

The study perturbs three tool-agent input sources to examine parameter filling: tool documents, user queries, and tool returns. Each source receives targeted changes that test different aspects of model behavior.

4 Methodology

  • The methodology perturbs tool documents, user queries, and tool returns to generate test samples for analyzing parameter failures.Initial correct behavior trajectories serve as test oracles.Source: §4 · p. 4

4.1 Tool Document Perturbation

  • Six tool-document perturbations test information loss, incorrect descriptions or types, and changes to parameter-description order or correspondence.They include removing descriptions or examples, substituting irrelevant descriptions, altering types, swapping descriptions, and reordering them.Source: §4.1 · pp. 4–5

4.2 User Query Perturbation

  • Four user-query perturbations test missing parameter information and robustness to complex or noisy queries.They remove the first or last parameter, replace parameters with complex phrases, or add interfering information.Source: §4.2 · p. 5

4.3 Tool Return Perturbation

  • Five tool-return perturbations test altered JSON keys, naming conventions, and malformed JSON handling.They fuzz or prefix keys, convert key naming styles, or corrupt the JSON format.Source: §4.3 · p. 5

5 Experiment

The experiments prepare ToolBench data and evaluate four LLMs using parameter-failure checks against oracle behavior. Each model contributes 600 initial trajectories, which are expanded with 15 perturbation methods for sensitivity analysis.

5.1 Data Preprocessing

  • After excluding unsolvable queries and scenarios without required parameters, GPT-4 augmentation supplements ToolBench data to yield 600 initial trajectories per LLM and 9,000 perturbed trajectories for sensitivity analysis.ToolBench includes single-tool and multi-tool commands; the 15 perturbation methods are applied to each LLM's 600 initial trajectories.Source: §5.1 · pp. 5–6

5.2 Setting

  • Experiments use four LLMs on an NVIDIA GeForce RTX A6000 GPU; failure rate is assessed against oracle behavior, with Rouge-L threshold 0.8 for Task Deviation and Specification Mismatch.The evaluated models are GPT-3.5-Turbo, GPT-4o-mini, ToolLLaMA-v2, and Qwen2.5-Plus; maximum_observation_length is 1024.Source: §5.2 · p. 6

6 Results

The results link parameter failures to information from user queries, tool documents, and tool returns, with different sources associated with distinct failure patterns. They also point to documentation accuracy, return formatting, error feedback, and parameter consistency as design concerns.

6 Results

  • Hallucination-name failures primarily reflect inherent LLM issues, while external information problems usually do not exacerbate them.Table 1 reports failure rates across base models and perturbations targeting tool documents, user queries, and tool returns.Source: §6 · p. 6, Table 1 · p. 7

6.1 Result on Tool Document Perturbation

  • WT perturbations particularly trigger task deviation and specification mismatch, while wrong types, missing parameters, and inaccurate returns can impair tool performance and usability.The authors recommend complete, accurate documentation, parameter-type verification, timely feedback, examples for underspecified queries, and regular evaluation of parameter use.Source: §6.1 · pp. 7–8

6.2 Result on User Query Perturbation

  • The two RP methods exceed 50% FR for task deviation on almost all LLMs, and removed query details can lead models to generate parameters that divert the execution trajectory.The authors also report that this deviation is more severe by Rouge-L and more likely to lead to task failure; they recommend query templates and protecting required information during transmission.Source: §6.2 · p. 8

6.3 Result on Tool Return Perturbation

  • Tool-return perturbations have lower FR than the other input sources but exceed 5% for some patterns; CF has the highest missing-information rate, and failures can disrupt downstream calls.The authors recommend unified return formats, better error messages, consistent parameter passing, and avoiding abrupt truncation of returned content.Source: §6.3 · p. 8

7 Conclusion

The conclusion identifies five parameter-failure patterns and connects failures in toolchain inputs with workflow disruption. It presents input structuring, adaptive error correction, and parameter-propagation attention as design priorities for reliable and traceable toolchains.

7 Conclusion

  • The paper distinguishes five failure patterns: missing required parameters, redundant information, hallucinated names, specification mismatch, and task deviation.These patterns respectively obstruct task processing, reduce return accuracy, prevent invocation, or reduce result value to the user.Source: §7 · p. 8
  • Ambiguous queries, defective tool documents, and poorly written returns can spread parameter failures through the toolchain and ultimately lead to task failure.Source: §7 · p. 8
  • The conclusion highlights proactive input structuring, adaptive error correction, and attention to parameter propagation as design priorities for reliable, traceable toolchain operations.Source: §7 · p. 8

Limitations

The study’s evidence is bounded by language, interaction setting, and tool type. The authors call for broader evaluation across multilingual, multi-turn, and non-API settings.

Limitations

  • Experiments focused mainly on English, so the observed taxonomy and failure-pattern relationships may differ across languages.Source: Limitations · p. 9
  • The evaluation used relatively controlled single-turn conversations and did not consider real-time or multi-turn scenarios.Source: Limitations · p. 9
  • The study focused on API-type invocations, leaving command-line tools and programming-language libraries unexplored.Source: Limitations · p. 9

Ethical Considerations

The paper flags privacy leakage from sensitive information used with tools and recommends sandboxing interactions involving local operating-system behavior.

Ethical Considerations

  • Tool use may expose personal sensitive information, and local operating-system interactions should run in an isolated sandbox to prevent direct exposure of sensitive system details.Source: Ethical Considerations · p. 9

A.1 Transfer Effect between Parameter Failure

Failure patterns often co-occur and transfer asymmetrically, while some query perturbations leave tool use unaffected when key information remains recoverable. The appendix also illustrates repeated invocation failure after an incorrect ID-format error.

A.1 Transfer Effect between Parameter Failure

  • More than half of failed cases across perturbations showed multiple failure patterns, and transfers were asymmetric, with redundant information especially likely to divert invocation from user intent.Except for parameter name hallucination, other failures showed a significant tendency to cause deviation; redundant information could alter tool results and lead to an incorrect subsequent chain of thought.Source: §A.1 · p. 10

A.2 Ineffective Perturbation Analysis

  • Removing a query parameter need not cause failure when the remaining sentence preserves its meaning or the agent can retrieve missing information with another tool.The paper’s example is the parameter “ja” in “Japanese language (ja)”; an agent may also invoke a tool to obtain missing information.Source: §A.2 · p. 10

A.4 Parameter Failure Cases

  • In an arXiv query example, a second invocation still returned an incorrect ID-format error after the agent attempted to revise the ID list.The first and subsequent tool returns both report “incorrect ID format.”Source: §A.4 · p. 17

Related papers

Loading 2507.15296v1…