PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models as Tool Makers

Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, Denny Zhou

arXiv:2305.17126v2cs.LGcs.AIcs.CLstat.ML

TL;DR

LATM addresses the need for suitable external tools by enabling LLMs to create reusable tools for problem solving. It separates powerful-model tool making from lightweight-model tool using and reports comparable performance to resource-intensive models at lower cost.

  • Problem

    Existing tool-using methods depend on suitable external tools, motivating a framework in which LLMs generate their own reusable tools for emerging tasks.

  • Method

    LATM uses a powerful LLM to create and verify reusable Python-function tools, then a lightweight LLM applies them to new task instances.

  • Results

    LATM performs comparably to resource-intensive models while being more cost-effective across various complex tasks, including challenging Big-Bench tasks.

  • Takeaways & Limitations

    LATM's one-time tool-making cost can be spread across repeated tool use, while its tool cache stores functionality for functionally analogous requests.

  • Takeaways & Limitations

    Real-world performance and safety may vary with task complexity, and validating generated tools in real-world settings remains challenging.

Abstract

from arXiv · show

Recent research has highlighted the potential of large language models (LLMs) to improve their problem-solving capabilities with the aid of suitable external tools. In our work, we further advance this concept by introducing a closed-loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs create their own reusable tools for problem-solving. Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks. 2) tool using: another LLM acts as the tool user, which applies the tool built by the tool maker for problem-solving. On the problem-solving server side, tool-making enables continual tool generation and caching as new requests emerge. This framework enables subsequent requests to access cached tools via their corresponding APIs, enhancing the efficiency of task resolution. Recognizing that tool-making requires more sophisticated capabilities, we assign this task to a powerful, albeit resource-intensive, model. Conversely, the simpler tool-using phase is delegated to a lightweight model. This strategic division of labor allows the once-off cost of tool-making to be spread over multiple instances of tool-using, significantly reducing average costs while maintaining strong performance. Furthermore, our method offers a functional cache through the caching and reuse of tools, which stores the functionality of a class of requests instead of the natural language responses from LLMs, thus extending the applicability of the conventional cache mechanism. We evaluate our approach across various complex reasoning tasks, including Big-Bench tasks. With GPT-4 as the tool maker and GPT-3.5 as the tool user, LATM demonstrates performance equivalent to using GPT-4 for both roles, but with a significantly reduced inference cost.

1 INTRODUCTION

LATM lets LLMs create reusable tools for new tasks, dividing tool making between a powerful model and tool using between a lightweight model. This reuse supports comparable performance at lower average cost and enables functional caching for analogous requests.

  • Motivation: LATM addresses the limited availability of suitable tools by enabling LLMs to generate reusable tools for new tasks.The framework introduces tool making and tool using as its two key stages.
  • Approach: A powerful model creates Python-function tools, while a lightweight model applies them to subsequent requests.This division assigns capability-intensive tool making to models such as GPT-4 and comparatively simpler tool use to models such as GPT-3.5 Turbo.
  • Efficiency: Tool making is performed once for a functionality, allowing the resulting tool to be reused across task instances and reducing average computational cost.The approach is motivated by recurring complex tasks such as scheduling meetings involving arithmetic reasoning.
  • Functional cache: LATM extends conventional response caching by storing tools for functionally analogous requests rather than natural-language responses.The functional cache is intended for streaming sequences of requests and can reduce average serving cost while maintaining high performance.
  • Results: Experiments on complex reasoning tasks, including challenging Big-Bench tasks, report performance on par with resource-intensive models at lower cost.The reported evaluation supports LATM's effectiveness across a range of tasks.

2 RELATED WORK

Related work augments LLM reasoning with programs, external tools, adaptive decoding, and model combinations. LATM differs by generating reusable tools and separating powerful tool making from lightweight tool use.

  • Chain of thought: Chain-of-thought prompting improves reasoning by representing reasoning traces in natural language or potentially programming-oriented forms.The related work positions CoT as an approach for enhancing complex-task problem solving.
  • Augmenting language models with tools: Prior systems supplement LLMs with task-specific actions and tools such as calculators, search engines, translators, and calendars.These methods combine language-model reasoning with external operations for complex tasks.
  • Augmenting language models with tools: Unlike methods that use Python executors for arithmetic substeps, LATM uses them to create reusable tools for other task instances.Separating tool maker and tool user also allows lightweight models to handle most inferences.
  • Adaptive generation: Speculative decoding accelerates generation with a faster model while a larger model scores its candidate tokens.LATM similarly passes artifacts from a more expensive model to a smaller model, but the artifact is a reusable tool.
  • Language model cascades: Language-model cascade research combines models to reduce costs or improve accuracy, whereas LATM identifies task categories and creates reusable tools.The distinction is between cascading model calls and transferring generated functionality.
  • Early attempts on tool-making: Early tool-making studies generated programs or abstract tools for individual problem-solving processes.LATM shares their aim of letting LLMs generate tools but emphasizes reusability and cost-effectiveness from division of labor.

3 LLM AS TOOL MAKER (LATM)

LATM separates one-time tool making from repeated tool using. A powerful model proposes, verifies, and wraps generic Python functions, while a lightweight model converts new queries into executable function calls.

  • Overview: LATM splits its pipeline into Tool Making and Tool Using, assigning different LLM types to balance performance and cost.The prompts used in the experiments are provided in Appendix C.
  • Tool Making: Tool Making uses a powerful model to create a generic, reusable Python function from a few task demonstrations.The stage is divided into tool proposing, verification, and wrapping.
  • Tool Proposing: Tool proposing generates a Python function from three demonstrations and retries after executable errors using the error history.This follows a programming-by-example paradigm.
  • Tool Verification: Tool verification generates and executes unit tests on three validation samples, then revises failed function calls in the tests.The procedure does not correct the function during test repair.
  • Tool Wrapping: Tool wrapping packages the function code with demonstrations showing how to convert task questions into function calls.If execution or verification fails beyond a preset threshold, Tool Making is considered failed.
  • Tool Using: Tool Using gives a lightweight model the verified tool and demonstrations, enabling it to generate function calls for new instances.The calls are executed, with optional postprocessing for formats such as multiple-choice answers.
  • Reuse: Because making is performed once per task type, the resulting Python tools can be reused across all instances, improving efficiency and cost-effectiveness.The tools provide a more generic form of Chain-of-Thought for algorithmic reasoning tasks.

4 LATM FOSTERS A FUNCTIONAL CACHE MECHANISM FOR LLM SERVING

LATM adds a dispatcher and repository of function APIs for sequential requests. Existing tools handle compatible instances through the tool user, while novel tasks trigger powerful-model tool creation or solving.

  • Dispatcher: A dispatcher LLM decides whether each incoming sequential task should use the tool user or tool maker.Its distinctive role is identifying tasks that existing tools cannot resolve and triggering appropriate tool generation.
  • Functional cache: The dispatcher searches a cache of tool-maker function APIs and assigns compatible instances and tools to the tool user.When no suitable tool exists, it identifies the instance as a novel task and can invoke a powerful model.
  • Illustration: Logical Deduction: For Logical Deduction, the tool maker creates a search function that enumerates five-object orderings and checks them against stated conditions.The tool user translates each natural-language question into conditions and function calls for each instance.

5 EXPERIMENTS

Experiments evaluate LATM across six diverse reasoning datasets, testing tool generation, lightweight tool use, dynamic-stream dispatch, model capacity, and CoT reuse. Results support reusable tools as a cost-effective way to maintain strong performance, while revealing capacity differences between tool-making and tool-using models.

  • 5.1 Experimental Setup: Experiments cover six datasets spanning Logical Deduction, Tracking Shuffled Objects, Dyck Language, Word Sorting, Chinese Remainder Theorem, and Scheduling Meeting.The first five datasets come from BigBench; Scheduling Meeting was constructed separately.
  • 5.2 Effectiveness of the Tool-Making Stage: GPT-4 frequently generates suitable algorithms, including permutation search for Logical Deduction, while tool verification mainly supplies function-call examples.Only 2 of 60 trials showed tool-maker self-correction guided by error messages.
  • 5.3 LATM Improves the Performance of Lightweight LLMs: LATM lets GPT-3.5 Turbo reach performance on par with GPT-4 while costing much less than GPT-4 inference across the evaluated tasks.GPT-4 creates the tool, and both GPT-3.5 Turbo and GPT-4 are evaluated as tool users; LATM significantly outperforms CoT prompting.
  • 5.4 Adapting LATM to a Dynamic Stream of Diverse Tasks: 95% ± 2% accuracy was achieved when the dispatcher identified suitable existing tools across 100 mixed-task samples.The evaluation used five random test-set constructions and six tasks.
  • 5.4 Adapting LATM to a Dynamic Stream of Diverse Tasks: 96% ± 3% accuracy was achieved when the dispatcher decided whether instances required new tool-making or existing tools.The 100-sample evaluation mixed unseen and existing tasks over multiple runs.
  • 5.5 Ablation Study: GPT-3.5 Turbo fails all five tool-making trials on hard tasks, whereas lightweight models can generate tools for simple tasks like Word Sorting.The authors identify insufficient generality and context-length constraints as tool-making failure factors, while GPT-3.5 Turbo offers the best tool-using performance-cost balance.
  • 5.5 Ablation Study: GPT-4-generated CoT performs similarly to human-written CoT and much worse than LATM on the two tested tasks.This ablation evaluates whether transferring natural-language reasoning traces can reproduce LATM's gains.

6 CONCLUSION AND FUTURE WORK

LATM enables LLMs to create and use reusable tools through separate Tool Making and Tool Using stages. The framework performs comparably to resource-intensive models while reducing computational costs, but evaluation is constrained by limited realistic interaction datasets.

  • 6 CONCLUSION AND FUTURE WORK: LATM combines Tool Making and Tool Using to let LLMs create and utilize tools for diverse tasks.The framework divides labor between stages and supports on-the-fly tool creation and usage through a dispatcher.
  • 6 CONCLUSION AND FUTURE WORK: LATM performs comparably to resource-intensive models while being more cost-effective across various complex tasks.
  • 6 CONCLUSION AND FUTURE WORK: The evaluation lacked high-quality datasets representing daily human-computer interactions in raw natural language.Examples include recurring tasks such as scheduling meetings or booking flights over email or phone calls.

A ILLUSTRATION OF THE DISPATCHER

In sequential online settings, the dispatcher checks incoming tasks against a cache of existing tools. It routes matched tasks to the tool user and unseen tasks to the tool maker for new tool creation.

  • A ILLUSTRATION OF THE DISPATCHER: The lightweight dispatcher assesses each incoming task instance in a sequential online setting.
  • A ILLUSTRATION OF THE DISPATCHER: When a suitable cached tool exists, the dispatcher selects it and forwards the task to the tool user.
  • A ILLUSTRATION OF THE DISPATCHER: When no suitable tool exists, the dispatcher routes the task to the tool maker to create one.

B BROADER IMPACT AND LIMITATIONS

LATM may expand the range and complexity of tasks LLMs handle while reducing computational resources and human intervention. However, autonomous tool creation raises safety, quality, control, and real-world validation concerns.

  • B BROADER IMPACT AND LIMITATIONS: LLM-generated tools could expand the range and complexity of tasks handled in customer service, technical support, and research and development.
  • B BROADER IMPACT AND LIMITATIONS: The framework could improve computational-resource efficiency and reduce human intervention for routine or repetitive tasks.
  • B BROADER IMPACT AND LIMITATIONS: Autonomous tool creation may produce suboptimal, incorrect, or harmful solutions and increase the risk of losing control without safeguards.
  • B BROADER IMPACT AND LIMITATIONS: LATM remains early-stage, with real-world performance and safety varying by task complexity and tool validation remaining challenging.

C LATM PROMPTS

LATM prompts instruct a model to identify or construct Python functions for task classes, verify them with unit tests, and convert function outputs into answers. The prompt set includes reusable APIs for several reasoning tasks.

  • C LATM PROMPTS: The prompt asks the model to write a generic Python function for the task and use standard Python libraries where possible.
  • C LATM PROMPTS: The verification prompt requires unit tests to check the function’s correctness.
  • C LATM PROMPTS: The solving format parses each question into function arguments, calls the function, and stores its return value in ret.
  • C LATM PROMPTS: For multiple-choice questions, the prompt converts ret into an answer option and stores it in ans.
  • C LATM PROMPTS: After verification succeeds, the history is summarized into a reusable function containing necessary imports and implementation.
  • C LATM PROMPTS: The square_dance API returns a dictionary representing final pair states after performing the specified switches.
  • C LATM PROMPTS: The prompt directs the model to inspect available functions and reply with the task name, or unknown when none applies.

D WRAPPED TOOLS

The paper wraps recurring reasoning patterns as reusable Python tools, then applies those tools to generated questions across several task classes. Examples cover logical deduction, shuffled-object tracking, sequence completion, sorting, modular arithmetic, and meeting scheduling.

  • Tool for Logical Deduction: Logical deduction tools search permutations of objects and retain orders satisfying all stated constraints.The wrapped function returns the option corresponding to the object occupying the queried position.
  • Tool for Tracking Shuffled Objects: The shuffled-object tool updates a dictionary by swapping partners sequentially, returning the final assignments.In the soccer example, the computed answer is right winger, option (C).
  • Wrapped Tools: Additional wrappers sort words alphabetically, search bounded integers satisfying divisor-remainder constraints, and find meeting slots from overlapping availabilities.The examples include a sorted word list and cases where no meeting slot satisfies the requested duration.

E DATASET CONSTRUCTION

The schedule-meeting dataset is generated from randomized meeting durations and availability intervals. Answers are computed from availability intersections by selecting the earliest interval long enough for the meeting, or returning no feasible slot.

  • Question Template: Each generated question asks for the best shared time slot between A and B, choosing the earliest when multiple slots work.The question template inserts the interval and both participants' availability lists.
  • Dataset Sampling: Meeting duration is randomly sampled from 0.5, 1, and 1.5 hours.Availability intervals are sampled between 8:00 and 18:00 at 30-minute granularity.
  • Answer Computation: The answer is the earliest intersection lasting at least the meeting duration, or “No time slot works.” when none exists.This computation directly determines the dataset label from the two availability sets.
Loading 2305.17126v2…