PaperlayerView exact revision on arXiv

Source-linked AI summary

Small LLMs Are Weak Tool Learners: A Multi-LLM Agent

Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, Fei Huang

arXiv:2401.07324v3cs.AIcs.CL

TL;DR

Small LLMs face limitations when a single model must handle the demands of tool learning. α-UMi distributes those demands across planner, caller, and summarizer LLMs and trains them progressively. Across model and data sizes, it outperforms the single-LLM approach in tool use.

  • Problem

    Tool learning requires planning, tool selection and invocation, and result summarization, but single-LLM approaches have performance limitations, especially with smaller models.

    Source: §1 · pp. 1–2

  • Method

    α-UMi assigns planning, tool invocation, and summarization to separate LLMs, trained first on the full task and then on role-specific datasets.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    Across various model and data sizes, α-UMi outperforms the single-LLM approach in tool use.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The modular design permits individual component updates and can use smaller LLMs for specialized capabilities.

    Source: §1 · p. 2

    The framework could be further optimized to improve its flexibility and applicability to a wider range of agent tasks.

    Source: §7 · p. 10

Abstract

from arXiv · show

Large Language Model (LLM) agents significantly extend the capabilities of standalone LLMs, empowering them to interact with external tools (e.g., APIs, functions) and complete various tasks in a self-directed fashion. The challenge of tool use demands that LLMs not only understand user queries and generate answers accurately but also excel in task planning, tool invocation, and result summarization. While traditional works focus on training a single LLM with all these capabilities, performance limitations become apparent, particularly with smaller models. To overcome these challenges, we propose a novel approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer. Each component is implemented by a single LLM that focuses on a specific capability and collaborates with others to accomplish the task. This modular framework facilitates individual updates and the potential use of smaller LLMs for building each capability. To effectively train this framework, we introduce a two-stage training paradigm. First, we fine-tune a backbone LLM on the entire dataset without discriminating sub-tasks, providing the model with a comprehensive understanding of the task. Second, the fine-tuned LLM is used to instantiate the planner, caller, and summarizer respectively, which are continually fine-tuned on respective sub-tasks. Evaluation across various tool-use benchmarks illustrates that our proposed multi-LLM framework surpasses the traditional single-LLM approach, highlighting its efficacy and advantages in tool learning.

1 Introduction

The paper argues that small single-LLM agents struggle to combine planning, tool invocation, and summarization, and introduces α-UMi to split these roles across LLMs. It pairs this design with GLPFT and reports better tool-use performance than the single-LLM approach across model and data sizes.

  • Tool learning requires planning, tool selection and invocation, and result summarization, while single-LLM approaches show performance limitations, especially with smaller models.Source: §1 · pp. 1–2
  • α-UMi assigns planning, tool invocation, and final-answer generation to dedicated planner, caller, and summarizer LLMs, which can be updated individually.Source: Figure 1 · p. 1, §1 · p. 2
  • GLPFT first fine-tunes a backbone on the full task, then continually fine-tunes planner, caller, and summarizer on role-specific datasets.Source: §1 · p. 2
  • Across various model and data sizes, α-UMi outperforms the single-LLM approach in tool use.Source: §1 · p. 2
  • The paper analyzes data scaling laws and investigates reasons for the framework’s improved performance.Source: §1 · p. 2

2 Related Works

Related work covers tool learning across diverse applications and LLM-powered agents across several task settings. The paper distinguishes its approach by assigning tool-learning capabilities to multiple LLMs rather than fine-tuning a single model.

2.1 Tool Learning

  • Tool-learning work applies external tools to real-time factual knowledge, multimodal tasks, code and math reasoning, and domain knowledge of AI models and APIs.Source: §2.1 · p. 2
  • Unlike prior single-LLM tool-learning approaches, this work targets smaller open-source LLMs by splitting capabilities among planner, caller, and summarizer roles.Source: §2.1 · pp. 2–3

2.2 LLM-powered Agents

  • LLM-powered agents address everyday tasks, explore Minecraft, and support software development and problem-solving.Source: §2.2 · p. 3
  • Agent techniques target reasoning and long-term memory, while recent open-source agent fine-tuning primarily trains a single LLM rather than a multi-LLM tool-learning agent.Source: §2.2 · p. 3

3 Methodology

α-UMi assigns tool-learning capabilities to dedicated planner, caller, and summarizer LLMs. GLPFT first trains a shared backbone on the whole task, then fine-tunes role-specific copies; the framework is designed to ease small LLMs’ workload.

3.1 Preliminary

  • In ReACT-style tool use, an LLM generates a rationale and action from the user instruction, system prompt, and prior execution trajectory.The trajectory records earlier rationales, actions, and tool observations; the final step produces an answer or abandons the run.Source: §3.1 · p. 3

3.2 The α-UMi Framework

  • α-UMi separates planning, tool invocation, and final-answer generation across dedicated planner, caller, and summarizer LLMs.The planner can activate the caller, transition to the summarizer, or give up; the caller generates tool requests, and the summarizer produces the final answer.Source: §3.2 · pp. 3–4

3.3 Global-to-Local Progressive Fine-Tuning

  • GLPFT first fine-tunes one backbone on the full tool-learning task, then copies it into planner, caller, and summarizer models for role-specific tuning.The local stage reorganizes the data by role and trains each model toward its rationale, action, or final-answer output.Source: §3.3 · pp. 4–5

3.4 Discussions

  • α-UMi is intended to ease small LLMs’ workload through task decomposition and role-specific prompts, while GLPFT bridges whole-task and subtask fine-tuning.The authors describe GLPFT as a contribution not found in existing multi-agent works.Source: §3.4 · p. 5

4 Experimental Settings

Experiments assess α-UMi on ToolBench and ToolAlpaca, including annotated-step and real-time API-call evaluations on ToolBench. The setup uses LLaMA-2-chat backbones and compares the framework with single- and multi-LLM baselines.

4.1 Benchmarks

  • ToolBench is split into in-domain and out-of-domain test instances by whether their tools appeared during training; ToolAlpaca and ToolBench evaluate API-based task completion.The paper also evaluates a program-aided mathematical-reasoning benchmark in Appendix D.Source: §4.1 · p. 5

4.2 Metrics

  • ToolBench uses annotated-step metrics and real-time RapidAPI evaluation, while ToolAlpaca is scored for process and final-answer correctness by GPT-4.Annotated-step metrics include Action EM, Argument F1, Rouge-L, API-name hallucination frequency, and planning accuracy.Source: §4.2 · p. 5

4.3 Implementation Details

  • The framework uses LLaMA-2-chat-7B/13B backbones; GLPFT trains the backbone for two epochs before role-specific fine-tuning.The planner and caller are fine-tuned for one epoch each and the summarizer for two; decoding is greedy with a maximum sequence length of 4096.Source: §4.3 · p. 6

4.4 Baselines

  • Baselines include a traditional Single-LLM, role-specific one-stage multi-LLM training, and a single model trained on combined role datasets.Comparisons also include zero-shot ChatGPT and GPT-4, and ToolLLaMA, a 7B LLaMA model fine-tuned on ToolBench.Source: §4.4 · p. 6

5 Results and Analysis

Across tool-use comparisons, α-UMi outperforms single-LLM and other multi-LLM baselines, with a 7B configuration sometimes exceeding a 13B single-LLM baseline. Scaling, training-loss, cost, and case-study analyses examine its performance patterns and trade-offs.

5.1 Overall Results

  • α-UMi surpasses ChatGPT and ToolLLaMA on all reported metrics except ToolAlpaca answer correctness, and beats Single-LLM on ToolBench and ToolAlpaca.The ToolBench comparison reports improvements particularly in planning, API calling, hallucination, and argument F1 metrics.Source: §5.1 · p. 6

5.2 Real-Time Test on ToolBench

  • +6.1 pass rate and +1.6 win rate versus ChatGPT, and +10.2 pass rate and +5.9 win rate versus ToolLLaMA, are reported for α-UMi (7B) on ToolBench real-time evaluation.Pass rate measures successful task completion; win rate compares solution paths with ChatGPT-ReACT.Source: Table 2 · p. 7, §5.2 · p. 7

5.1 Overall Results

  • A 7B α-UMi outperforms the 13B Single-LLM baseline and also exceeds Multi-LLMone-stage and Single-LLMmulti-task.The authors attribute the latter comparisons to GLPFT mitigating the difficulty of assigning all roles to limited-capacity models.Source: §5.1 · p. 7

5.3 Data Scaling Law

  • Across 12.1k, 15.7k, 31.3k, 47.0k, and 62.7k training instances, α-UMi improves consistently with scale and exceeds Single-LLM especially on Plan ACC, Act. EM, Hallu., and Arg. F1.R-L gains are minor; Multi-LLMone-stage and Single-LLMmulti-task fluctuate on most metrics, while Single-LLM peaks at different scales across metrics.Source: Figure 4 · p. 8, §5.3 · p. 8

5.4 Why α-UMi Works?

  • After two epochs, Single-LLM loss nearly stagnates, while α-UMi continues reducing rationale and action losses, which the authors link to surpassing Single-LLM’s performance upper bound.Rationale, action, and answer losses all decline during the initial two epochs.Source: Figure 5 · p. 8, §5.4 · pp. 8–9

5.5 Cost of α-UMi

  • 3× storage, 1.3× training compute, and 1.5× training duration are reported for α-UMi versus same-size Single-LLM backbones, with similar inference time and training GPU memory cost.The authors also report that 7B α-UMi costs less in training and inference than 13B Single-LLM while outperforming it.Source: Table 3 · p. 9, §5.5 · p. 9

5.6 Case Study

  • In composite tasks, α-UMi plans aligned API steps where Single-LLM may misinvoke APIs or loop; a case study also shows recovery by switching from a broken API to an alternative.Source: §5.6 · p. 9

6 Conclusion

The paper addresses the challenge of equipping a small LLM with the capabilities required for tool learning. It presents α-UMi and GLPFT, reporting that their combination improves tool-learning performance and can let small LLMs surpass a larger single-LLM agent.

6 Conclusion

  • The paper targets the challenge of training one small LLM to acquire the extensive abilities required for a tool-learning agent.Source: §6 · pp. 9–10
  • α-UMi delegates tool learning to three small LLMs serving as planner, caller, and summarizer, trained with global-to-local progressive fine-tuning.Source: §6 · p. 10
  • Across four tool-learning benchmarks, the authors conclude that task decomposition and progressive fine-tuning enhance the system’s upper-bound ability, with small LLMs potentially surpassing a larger single-LLM agent.Source: §6 · p. 10

7 Limitations

The limitations section identifies future work in combining small models with a powerful closed-source LLM and improving the framework’s flexibility and applicability to more agent tasks.

7 Limitations

  • The authors identify large-plus-small collaboration and broader agent-task flexibility as future directions for the framework.They specifically suggest integrating small LLMs with GPT-4 and further optimizing flexibility and applicability.Source: §7 · p. 10

8 Ethical Statement

The section states that the framework uses public tool-learning benchmarks without ethically risky training data, and provides prompts for its agent roles. The prompts cover ToolBench and ToolAlpaca as well as math tasks.

8 Ethical Statement

  • The framework uses public ToolBench and ToolAlpaca data, has no training on data posing ethical risks, and uses a safety-aligned LLaMA-2-chat backbone.Source: §8 · p. 10

A.1 Pplan for ToolBench and ToolAlpaca

  • The ToolBench and ToolAlpaca planner prompt supplies available APIs and conversation history, then asks the planner to choose the next step and whether caller, conclusion, or termination follows.Source: §A.1 · p. 12

A.2 Pcall for ToolBench and ToolAlpaca

  • The ToolBench and ToolAlpaca caller prompt uses available APIs, history, and the planner’s thought to specify an API action and its input.Source: §A.2 · p. 12

A.3 Psum for ToolBench and ToolAlpaca

  • The ToolBench and ToolAlpaca summarizer prompt asks for a conclusion based on conversation history.Source: §A.3 · p. 12

A.4 Pplan for MATH and GSM8K

  • For math tasks, the planner prompt frames step-by-step problem solving with Python and asks whether the caller should generate code or the conclusion should answer.Source: §A.4 · p. 12

A.5 Pcall for MATH and GSM8K

  • For math tasks, the caller prompt receives the problem and execution logs, considers the current step’s thought, and generates code.Source: §A.5 · pp. 12–13

A.6 Psum for MATH and GSM8K

  • For math tasks, the summarizer prompt asks for a conclusion based on the problem and historical execution logs.Source: §A.6 · p. 13

B Details of Benchmarks

The benchmark details describe ToolBench’s API-call data, its in-domain and out-of-domain test split, and a step-level evaluation designed to reduce problems from changing live APIs.

B.1 ToolBench

  • ToolBench provides 62,694 sampled training trajectories from 125,387 trajectories covering 16,464 RapidAPI APIs, with 4.1 execution steps per trajectory on average.Source: §B.1 · p. 13
  • ToolBench’s six test groups yield 400 in-domain instructions using training tools and 200 out-of-domain instructions using unseen tools.Each group contains 100 instructions; groups ending in “instruction” are in-domain, while “tool” or “category” groups are out-of-domain.Source: §B.1 · p. 13
  • To limit unfairness from changing or failing APIs, evaluation compares generated rationale and action at each step against annotated GPT-4 outputs using GPT-4’s prior trajectory.Source: §B.1 · p. 13

B.2 ToolAlpaca

ToolAlpaca evaluates API calling with ChatGPT-mocked APIs and results, using a simulator and GPT-4 judgments of execution and final answers.

B.2 ToolAlpaca

  • ToolAlpaca uses APIs and API results mocked from ChatGPT rather than live APIs, and evaluates tool use in a simulator.Source: §B.2 · p. 13
  • ToolAlpaca contains 4,098 training instances averaging 2.66 execution steps and a test set of 100 user instructions.Source: §B.2 · p. 13
  • GPT-4 judges whether the agent’s execution matches the pre-generated reference process and whether its final answer solves the instruction.Source: §B.2 · p. 13

C Static Evaluation on ToolBench

ToolBench static evaluation assesses an agent step by step using the ground-truth prior trajectory. By isolating each step, it avoids error propagation from external execution problems and complements real-time evaluation.

C Static Evaluation on ToolBench

  • At each step, the agent receives the ground-truth previous trajectory and generates the current rationale and action.Metrics compare those outputs with the annotated ground truth.Source: C Static Evaluation on ToolBench · p. 13
  • Using the previous ground-truth trajectory at each step prevents error propagation from network blocks, API breakdowns, or exhausted quotas, supporting fairer comparisons.The authors describe static evaluation as an effective complement to real-time evaluation.Source: C Static Evaluation on ToolBench · p. 13

D.1 MATH and GSM8K

The appendix reports that α-UMi outperforms baselines on MATH and GSM8K and describes its program-aided setup and illustrative cases.

D.1 MATH and GSM8K

  • The program-aided agent assigns planning rationales to the planner, calculation programs to the caller, and final-answer synthesis to the summarizer.Training trajectories were collected with GPT-3.5-turbo-1106 and GPT-4, filtering out trajectories that did not yield correct final answers.Source: §D.1 · p. 14
  • 5,536 ChatGPT and 573 GPT-4 trajectories were collected for MATH, while GSM8K contributed 6,213 ChatGPT trajectories.The trajectories were gathered from the datasets’ training sets.Source: §D.1 · p. 14
  • α-UMi outperforms the baselines on MATH and GSM8K in final-answer accuracy.The test sets contain 5,000 MATH problems and 1,319 GSM8K problems.Source: §D.1 · p. 14
  • In the case studies, α-UMi finishes a task in two steps when tools are specified and redirects after a video-search API is found broken.The planner directs the caller to try an alternative API, after which the task is resolved.Source: Figure 6 · p. 15, Figure 7 · p. 15
Loading 2401.07324v3…