PaperlayerView exact revision on arXiv

Source-linked AI summary

RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning

Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, Xuanjing Huang

arXiv:2401.08326v3cs.CLcs.AI

TL;DR

Tool-learning research has largely overlooked LLM stability under real-world noise. RoTBench evaluates robustness across five noise environments and three tool-use stages, and RoTTuning improves robustness through greater environmental diversity. The experiments reveal substantial noise sensitivity, including a GPT-4 drop from 80.00 to 58.10 and reduced GPT-family performance under mild noise.

  • Problem

    Existing tool-learning studies largely overlook LLM robustness to inevitable noise in real-world environments.

    Source: §1 · p. 2

  • Method

    RoTBench evaluates tool learning across five noise environments and three stages, while RoTTuning increases training-environment diversity.

    Source: §1 · p. 2

  • Results

    Experiments show strong noise sensitivity, with GPT-4 dropping from 80.00 to 58.10 and GPT-family noise correction hindering performance under mild noise.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Robustness in noisy tool learning requires explicit evaluation and training across diverse environments.

    Source: §1 · p. 2, §6 · p. 9

    The study focuses on single-round tool use and does not examine whether LLMs self-correct in response to environmental feedback.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Tool learning has generated widespread interest as a vital means of interaction between Large Language Models (LLMs) and the physical world. Current research predominantly emphasizes LLMs' capacity to utilize tools in well-structured environments while overlooking their stability when confronted with the inevitable noise of the real world. To bridge this gap, we introduce RoTBench, a multi-level benchmark for evaluating the robustness of LLMs in tool learning. Specifically, we establish five external environments, each featuring varying levels of noise (i.e., Clean, Slight, Medium, Heavy, and Union), providing an in-depth analysis of the model's resilience across three critical phases: tool selection, parameter identification, and content filling. Experiments involving six widely-used models underscore the urgent necessity for enhancing the robustness of LLMs in tool learning. For instance, the performance of GPT-4 even drops significantly from 80.00 to 58.10 when there is no substantial change in manual accuracy. More surprisingly, the noise correction capability inherent in the GPT family paradoxically impedes its adaptability in the face of mild noise. In light of these findings, we propose RoTTuning, a strategy that enriches the diversity of training environments to bolster the robustness of LLMs in tool learning. The code and data are available at https://github.com/Junjie-Ye/RoTBench.

1 Introduction

Prior tool-learning research largely evaluates LLMs in stable environments, leaving robustness to real-world noise insufficiently examined. RoTBench measures this robustness across noise levels and tool-use stages, while RoTTuning improves adaptability through environmental diversity.

Research gap

  • Current studies assess tool usage mainly in controlled environments, overlooking robustness to inevitable real-world noise.This gap matters because tool names can change without changing functionality, yet such changes may impede tool use.Source: §1 · pp. 1–2

RoTBench

  • RoTBench evaluates LLM robustness across five noise environments and three tool-learning stages.The environments are Clean, Slight, Medium, Heavy, and Union; the stages are tool selection, parameter identification, and content filling.Source: §1 · p. 2

Empirical findings

  • GPT-4 performance drops from 80.00 to 58.10 despite no substantial change in manual accuracy.Experiments across six widely used LLMs show marked sensitivity to noise.Source: §1 · p. 2
  • The GPT family’s inherent noise-correction capability appears to hinder performance in mildly noisy environments.This reveals a tension between models’ inherent capabilities and robustness.Source: §1 · p. 2

RoTTuning

  • RoTTuning improves robustness by increasing training-environment diversity, achieving an average gain of 16.10 points across diverse environments.The method is designed to improve adaptability across a wide range of environments.Source: §1 · p. 2

2 Related Work

Related work spans tool-learning evaluation and general robustness testing, but existing efforts primarily study controlled environments. RoTBench addresses this omission by evaluating robustness under noisy tool-learning conditions.

Analysis of Tool Learning

  • Tool-learning research has progressed from outcome evaluation to analysis of the broader tool-use process and specialized model training.Prior work examines tool-selection accuracy, generated responses, process capabilities, and limitations.Source: §2 · p. 2

Research gap

  • Existing tool-learning efforts mainly use controlled environments, motivating robustness evaluation in real-world scenarios.Because tool-learning outcomes can affect the physical world, the paper argues that robustness requires thorough evaluation.Source: §2 · pp. 2–3

Robustness Testing of LLMs

  • General robustness research evaluates LLM consistency under noisy inputs, including benchmarks that inject noise into prompts.These studies establish robustness testing as relevant to practical deployment.Source: §2 · pp. 2–3

3 RoTBench

RoTBench evaluates tool-learning robustness across noisy environments, data construction choices, and the stages of tool use. It combines five noise settings with staged metrics for tool selection, parameter identification, and content filling.

3 RoTBench

  • RoTBench evaluates robustness across five noise environments and three tool-learning stages.The environments are Clean, Slight, Medium, Heavy, and Union; evaluation covers tool selection, parameter identification, and content filling.Source: §3 · p. 3, §3.2 · p. 3, §3.3 · p. 4

3.1 Data Collection

  • The dataset uses seven real-world application scenarios with 15 user queries each, whose tool invocation paths were manually labeled.The data are drawn from ToolEyes and standardized for evaluation.Source: §3.1 · p. 3

3.2 Environments Construction

  • The benchmark derives noisy environments from 105 Clean-level test cases using character edits, reversals, nonsense names, shuffled names, added parameters, and combined perturbations.Slight and Medium each contain 210 test cases, Heavy contains 210, and Union contains 105 cases affecting both tool names and parameters.Source: §3.2 · pp. 3–4

3.3 Staged Evaluation

  • Evaluation defines separate scores for tool selection, parameter identification, and content filling, assessing suitable tool choice, required-parameter identification, and the population of parameter contents for each given test case.Source: §3.3 · pp. 4–5

4 Experiments

Experiments evaluate six widely used LLMs across noisy environments and tool-learning stages, revealing substantial robustness gaps and a GPT-family failure mode under slight noise.

4.1 Model Selection

  • The experiments test four open-source and two closed-source models, focusing on their initial tool use because multi-turn intermediate trajectories are difficult to compare.The evaluated closed-source models are GPT-3.5-turbo and GPT-4; the open-source models are ToolLLaMA-2-7B-v1/v2 and NexusRaven-13B-v1/v2.Source: §4.1 · p. 5, §4.2 · p. 6

4.2 Main Results

  • 20.32 points: average LLM tool-selection performance decreases from Clean-level to Union-level, versus a 2.86-point decrease for humans.The results show substantial performance fluctuations for LLMs across environments while human performance remains relatively stable.Source: §4.2 · p. 6
  • Tool-name noise affects LLM performance more strongly than parameter noise throughout tool learning, while Union-level performance exceeds Heavy (Tool)-level performance.Parameter noise has minimal impact during tool selection and less influence on later stages than tool-name noise.Source: §4.2 · p. 6
  • 22.91 points: GPT-4’s average performance improves with two interactive examples, but cross-environment standard deviation rises from 8.14 to 12.56.Interactive examples improve performance during the third interaction turn without improving robustness.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 Why do GPT family of models NOT perform well in Slight-level environment?

  • GPT-family performance is lower in Slight-level than Medium-level environments because noise correction changes noisy tool names into incorrect clean names.For example, correcting “predOict_aTge” to “predict_age” produces an erroneous output; noise-correction errors exceed one-third for GPT-3.5-turbo in relevant stages.Source: §4.3 · p. 7

5 RoTTuning

RoTTuning increases training-environment diversity through four phases and produces RoTLLaMA, whose performance is more stable across noisy environments than GPT-4.

5 RoTTuning

  • RoTTuning uses query expansion, trajectory generation, environment augmentation, and generalizability training to increase environmental diversity.The four phases are illustrated in Figure 5.Source: §5.1 · p. 7, §5 · p. 7

5.1 Method

  • 4,077 new queries and 12,247 training examples are generated before augmentation, producing 22,747 trajectories across Clean, Slight, Medium, Heavy, and Union conditions.Query expansion filters redundant examples; trajectory generation limits tool use to nine turns; augmentation adds randomized noisy-environment data.Source: §5.1 · p. 8
  • RoTLLaMA is obtained by five-epoch LoRA fine-tuning of LLaMA-2-7B-base with position interpolation extending context length to 8096.The method uses diverse trajectories generated across environmental conditions.Source: §5.1 · p. 8

5.2 Experimental Results

  • 12.38, 16.19, and 14.76: RoTLLaMA’s extreme performance differences across tool selection, parameter recognition, and content filling are below GPT-4’s 21.90, 20.95, and 20.95.These results indicate greater cross-environment performance stability for RoTLLaMA across all three evaluated phases.Source: §5.2 · p. 8, Table 5 · p. 8
  • Removing environment augmentation decreases mean performance and increases within-environment standard deviation, whereas removing LoRA slightly decreases performance without substantially changing robustness.Removing both components causes a 16.10-point performance degradation.Source: §5.2 · pp. 8–9, Figure 6 · p. 8

6 Conclusion

The paper introduces RoTBench to evaluate robustness in noisy tool learning and RoTTuning to improve robustness through greater training-environment diversity.

6 Conclusion

  • RoTBench evaluates LLM robustness across five noise-level environments, while RoTTuning improves robustness by increasing training-environment diversity.The benchmark and training approach address robustness in tool learning.Source: §6 · p. 9

Limitations

The paper limits its robustness analysis to single-round tool use and to noise in tool and parameter names, while documenting the open-source models included in evaluation.

Limitations

  • The benchmark assesses robustness in a single tool-use round and does not examine self-correction from environmental feedback.For GPT-4, analysis of the first two interaction rounds did not improve robustness.Source: Limitations · p. 9
  • The analysis focuses on noise in tool and parameter names rather than the meanings conveyed by tool documentation.The authors motivate this choice by reporting that models primarily rely on tool and parameter names.Source: Limitations · p. 9

A Details of LLMs

  • Four open-source models were selected for evaluation, including ToolLLaMA-2-7B-v1, ToolLLaMA-2-7B-v2, NexusRaven-13B-v1, and NexusRaven-13B-v2.The selected models use different training data and tool-calling approaches, including code nesting and parallel function calls.Source: A Details of LLMs · p. 12, §A.1 · p. 12

A.2 Closed-Source LLMs

The evaluation includes two closed-source models from the GPT family: GPT-3.5-turbo and GPT-4.

A.2 Closed-Source LLMs

  • The study evaluates two representative closed-source models from the GPT family.They are included alongside four open-source models.Source: §A.2 · p. 12
  • GPT-3.5-turbo is described as a conversational model with strong natural-language and tool-invocation capabilities.The paper characterizes it as the most potent and cost-efficient model in the GPT-3.5 series.Source: §A.2 · p. 12
  • GPT-4 is described as OpenAI’s most robust LLM, with formal multimodal-input support and broader social-requirement coverage.The paper also describes GPT-4 as safer and more beneficial than its predecessor.Source: §A.2 · p. 12

B Experimental Setup

The experiments use model-specific inference templates, human benchmarks, and binary stage-level scoring for tool selection, parameter identification, and content filling.

B Experimental Setup

  • Inference uses ReAct prompts for the ToolLLaMA-2-7B and GPT families, while NexusRaven models use their official nested-function template.The templates differ because NexusRaven outputs nested functions.Source: B Experimental Setup · p. 12
  • Three university students independently answer the same tasks using identical documentation, and their average score provides the human benchmark.Human performance is measured across environments with different noise levels.Source: B Experimental Setup · p. 12
  • Each data point receives a binary score at every stage because tool calls either succeed or fail, and small errors can invalidate execution.Errors in tool selection, parameter identification, or content filling can each cause failure or undesirable results.Source: B Experimental Setup · p. 12

C Results in Different Scenarios

Results vary across scenarios, models, and noise levels: sequential tool dependencies and weak noise correction create failures, while model-specific training produces unexpected performance patterns.

C Results in Different Scenarios

  • Tool-call ordering can cause errors when obtaining a parameter requires a prior tool execution, such as searching locations before listing properties.This failure reflects dependencies among sequentially invoked tools.Source: C Results in Different Scenarios · p. 13
  • LLMs perform worse in slight-noise information-retrieval environments than in medium- and heavy-noise environments because of limited noise-correction capability.The paper reports that model-perceived environmental complexity can diverge from human intentions.Source: C Results in Different Scenarios · p. 13
  • ToolLLaMA-7B-v1 scores 20 in the clean environment and 40 in the union environment for application manipulation, while GPT-4 repeatedly requests unnecessary API keys.The reported disparity is attributed to ToolLLaMA-7B-v1 performing better when only two tools plus “ask_to_user” and “finish” are available.Source: C Results in Different Scenarios · p. 13

D Examples for Noise Correction

  • Table 13 presents examples of noise correction by GPT-family models during tool selection and parameter identification.These examples connect the reported scenario-level behavior to the two earlier decision stages.Source: D Examples for Noise Correction · p. 13

E Further Studies about RoTTuning

Further studies show that RoTTuning generalizes across environments, reduces tool hallucinations, and improves tool learning when applied to ToolLLaMA-2-7B. The appendix also documents the inference, query-expansion, and trajectory-generation prompts used in evaluation.

E Further Studies about RoTTuning

  • RoTTuning trained on data from a single environment achieves superior performance in other environments, demonstrating robust generalization.The comparison is against models trained without augmentation, as reported for Table 14.Source: E Further Studies about RoTTuning · p. 13, Table 14 · p. 18
  • RoTTuning produces significantly fewer tool hallucinations than the GPT family across environments, while NexusRaven models also show low hallucination counts.The authors connect this result to reduced interference from noise and note code-based tool learning as a viable direction.Source: E Further Studies about RoTTuning · p. 13, Table 15 · p. 19
  • Fine-tuning ToolLLaMA-2-7B with RoTTuning trajectories improves tool learning across all stages and strengthens overall robustness.The reported stage-wise performance extremes are 12.33/13.33/9.53 for RoTToolLLaMA versus 26.67/16.67/10.95 for ToolLLaMA-2-7B-v2.Source: E Further Studies about RoTTuning · p. 13, Table 16 · p. 19

F Prompt Template for Inference

G Prompt Template for Query Expansion

Related papers

Loading 2401.08326v3…