Source-linked AI summary
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, Ying Shan
TL;DR
Open-source LLMs lack an efficient, accessible way to use multimodal tools, while proprietary approaches depend on costly models and inaccessible data. GPT4Tools generates multimodal instruction data from an advanced teacher and applies LoRA to open-source models, with a benchmark for tool use. The resulting models improve seen-tool invocation and achieve strong zero-shot use of unseen tools, including performance comparable to GPT-3.5.
Problem
Proprietary LLMs provide multimodal tool-use capabilities but rely on prohibitive computation, inaccessible specialized data, and large amounts of data for tool-equipped agents.
Method
GPT4Tools prompts advanced teachers with visual content and tool descriptions to generate instruction data, then uses LoRA to fine-tune open-source LLMs and benchmarks tool use.
Results
GPT4Tools improves seen-tool invocation and enables unseen-tool use, with fine-tuned Vicuna-13B gaining 9.3% absolute successful rate over GPT-3.5 and performing comparably on unseen tools.
Takeaways & Limitations
The approach equips open-source LLMs with multimodal tools for visual comprehension and image generation while supporting tool-use evaluation across decisions, tools, and arguments.
Takeaways & Limitations
Success rates remain below 100%, and verbose fixed prompts for explicit tool invocation reduce computational efficiency.
Abstract
from arXiv · showhide
This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools. It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools. The code and demo are available at https://github.com/StevenGrove/GPT4Tools.
1 Introduction
Existing multimodal tool-use agents depend heavily on proprietary models, costly computation, and specialized data. GPT4Tools addresses this gap by self-instructing open-source LLMs with multimodal contexts and evaluates their tool-use ability across tasks.
- Proprietary LLMs offer strong zero-shot reasoning but require high computational cost and data that are not publicly accessible.
- Prior multimodal agents rely on sophisticated prompt engineering and GPT-3.5 for instruction decomposition, limiting accessibility and further advancement.
- GPT4Tools prompts advanced teachers with visual content and tool descriptions to generate instruction-following data, then applies LoRA to open-source models.
- The method targets visual comprehension and image-generation problems, including grounding, segmentation, image instruction, and visual question answering.
- GPT4Tools introduces a benchmark for tool-use effectiveness across diverse tasks, evaluated in zero-shot and fine-tuning settings.
- Fine-tuned Vicuna-13B gains 9.3% absolute successful rate over GPT-3.5 and performs comparably to GPT-3.5 on unseen tools.
2 Related Work
Related work spans multimodal language models, instruction tuning, and language-model tool use. GPT4Tools instead equips language models with specialized multimodal tools through self-instruction.
- Vision and Language Model: Vision-language research represents images textually or projects image features into textual space, while generalist models jointly process images and text.
- GPT4Tools: Figure 1 depicts GPT4Tools generating tool-related instruction data from ChatGPT using image content and tool definitions, followed by LoRA training of an open-source LLM.
- Vision and Language Model: GPT4Tools uses specialized multimodal tools for distinct vision tasks, promoting scalability and avoiding forgetfulness from repeated fine-tuning.
- Instruction Tuning: Instruction tuning improves language-model task following, while Self-Instruction reduces reliance on human-written instructions by using model-generated instructions.
- Tool Usage: Earlier tool-use studies used retrieval, calculators, translators, calendars, or few-shot prompting; ToolFormer self-instructed GPT-J with five tools.
- Tool Usage: GPT4Tools uses GPT-3.5 as a teacher to distill off-the-shelf language models for access to many visual models.
3 Method
GPT4Tools constructs multimodal tool-use instructions with an advanced teacher, augments them with negative and contextual examples, and tunes open-source language models with LoRA. Its evaluation separately measures decision, tool-name, argument, and end-to-end chain accuracy across seen and unseen tools.
- Dataset Construction: GPT4Tools prompts ChatGPT with image content and tool descriptions to generate tool-related instruction-following data.Image content is constructed from captions and bounding boxes, grounding generated instructions in visual inputs.
- Dataset Construction: Image-conditioned generation produces more informative and diverse instructions than generation without image content.The paper attributes this difference to variation in image content, whereas unconditioned instructions show greater similarity.
- Dataset Construction: Filtering 70K raw items yields 41K retained items formatted as instruction-response examples containing tool thoughts, actions, arguments, and observations.The response format represents each action with an Action and Action Input, followed by the tool name and arguments.
- Dataset Construction: The dataset combines positive, negative, and contextual samples so models learn both when to invoke tools and how to call them in context.Negative samples reduce unnecessary tool invocation, while multi-turn context samples support tool calls within ongoing conversations.
- Instruction Tuning: LoRA tuning freezes the language model and optimizes rank-decomposition components using the original autoregressive objective.The target-response probability is computed over a sequence conditioned on image content, instruction tokens, and preceding response tokens.
- Evaluation Approach: The evaluation uses separate successful rates for thought, action, arguments, and complete action chains, with validation on 23 tools and testing on 8 novel tools.A complete chain requires correct thought, tool name, and arguments; two consecutive actions both must be correct for a 100% successful rate.
4 Experiments
Experiments evaluate GPT4Tools across language models, seen and unseen tools, data augmentations, tuning iterations, and visual case studies. Fine-tuning substantially improves tool invocation and supports generalization to tools absent from training.
- Implementation Details: The study uses ChatGPT to generate raw instruction-following data, with 71K pairs based on 23 Visual ChatGPT tools, and tunes LLaMA, Vicuna, and OPT using LoRA.The evaluation includes a validation set with seen tools and a test set containing unseen tools.
- Main Result: 93.2% SR is achieved by fine-tuned OPT-13B, rising from 0% without GPT4Tools on the validation set.The reported comparison shows substantial changes in tool invocation capability after fine-tuning.
- Main Result: 90.6% SR on new tools is achieved by fine-tuned Vicuna-13B, comparable to GPT-3.5 and demonstrating unseen-tool invocation after fine-tuning.Fine-tuned LLaMA-13B also improves its new-tool SR by 67.9% over the original model.
- Ablation Study: 94.1% SR is obtained with image content, context samples, and negative samples, compared with 36.9% without image-conditioned instructions.The ablation attributes gains to more diverse instructions, contextual action partitioning, and improved discrimination of when to invoke tools.
- Ablation Study: After iteration 800, SRt, SRact, SRargs, and SR steadily improve, whereas performance fluctuates substantially between iterations 400 and 800.The pattern is interpreted as progressive adaptation to the generated dataset.
- Case Study: GPT4Tools produces an accurate edge-conditioned image, while Visual ChatGPT produces a weakly correlated image and LLaVa returns only a caption.A fine-tuned Vicuna-13B also completes visual commands by invoking visual tools.
5 Limitation
GPT4Tools remains imperfect and uses verbose, fixed prompts for explicit tool invocation, motivating more efficient implicit tool use.
- Limitation: GPT4Tools has success rates below 100%, so further improvement is needed for practical applications.The authors identify this as a limitation of the current approach.
- Limitation: Its verbose, fixed invocation prompt reduces computational efficiency, motivating future work on implicit tool invocation.Attention-based architectures compute relationships between all tokens, making complex prompts costly.
6 Conclusion
GPT4Tools enables open-source LLMs to use multimodal tools through instruction data generated from ChatGPT and LoRA optimization. Its benchmark evaluates tool-use decisions and arguments, with tuned models performing comparably to GPT-3.5 on unseen tools.
- GPT4Tools constructs tool-related instructional data from advanced ChatGPT and augments it with negative and context samples.The method uses the resulting dataset to train language models for multimodal tool usage.
- LoRA optimization enhances LLMs’ tool-usage capabilities for visual comprehension and image generation.
- The benchmark evaluates when to use tools, which tools to use, and the arguments supplied to invoked tools.
- Tuned LLMs perform comparably to GPT-3.5 on unseen tools.
A GPT4Tools Dataset
The GPT4Tools dataset summarizes its tool inventory and sample distribution. Figure 7 distinguishes negative samples from positive samples using color.
- Table 5 summarizes the tool names included in GPT4Tools.Gray names come from Visual ChatGPT, while black names are new in GPT4Tools.
- Figure 7 shows the data distribution of GPT4Tools.
- Purple denotes negative samples in Figure 7, while the other colors denote positive samples.
A.1 Training Set
The training set contains tool-use instruction-following data covering 23 tools. Its distribution is illustrated in Figure 7, and the set trains language models to invoke tools.
- The training set contains 71.4K instruction-following data items, including 35.7K items using tools.
- The training data originates from 41K items in Y + S, with some actions requiring two tools.
- The instructional data covers 23 tools, whose names are listed in Table 5.
- The training set is used to instruct the language model to invoke tools.
A.2 Evaluation Set.
GPT4Tools evaluates tool usage with validation and test sets, supports 31 multimodal tools, and generates and prompts tool-related interactions using visual content.
- The evaluation set comprises a validation set and a test set.
- The validation set contains 1170 samples using the training tools and evaluates tool understanding after fine-tuning.Each tool appears almost 50 times, and the set includes augmented samples.
- The test set contains 652 single-turn samples with 8 tools unseen during training, evaluating zero-shot tool invocation.
- GPT4Tools supports 31 tools, including 23 from Visual ChatGPT and 8 new tools dependent on image generation and understanding models.
- GPT-3.5 generates instruction-following data from visual captions and bounding boxes, after which noisy instructions are filtered and augmented.
- During tool use, predicted image captions replace <image content> when the user input requires image content as a precondition.
C Case Study
The case study identifies two failure modes in tool-use data and control: generated instructions contain multiple kinds of noise, while GPT-3.5 can invoke incorrect tools.
- Generated instructions can contain format errors, argument errors, and incorrectly assigned tools.These cases are reported as noise in data generated by ChatGPT or GPT-4.
- A practical filtering step is necessary before using language-model-generated instructions.
- GPT-3.5 sometimes invokes the wrong tools in response to user commands.The paper presents incorrect examples in Tables 10 and 11.
D Experiment Settings
The experiments benchmark tool usage with a self-built dataset and specify inference prompts, incorrect GPT-3.5 examples, and fine-tuning configuration through accompanying tables.
- Tool-usage ability is benchmarked using a self-built dataset, with fine-tuning configuration recorded in Table 12.
- During inference, image content is replaced by image-caption results and user input is filled with the user command.
- Tables 10 and 11 provide incorrect examples from GPT-3.5, specifically text-davinci-003.
- Table 12 reports the fine-tuning configuration.