Source-linked AI summary
Toolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom
TL;DR
Language models struggle with limitations such as factual lookup and precise calculation, while existing tool-use methods require extensive annotation or remain task-specific. Toolformer learns to use external APIs self-supervised and considerably improves zero-shot performance, even outperforming GPT-3 across varied downstream tasks.
Problem
Language models remain limited in up-to-date information access, factual accuracy, mathematical calculation, and time awareness, while existing tool-use methods require extensive annotation or task-specific designs.
Method
Toolformer learns to use a variety of external tools in a self-supervised way, with independently generated API calls used for finetuning.
Results
Toolformer considerably improves zero-shot performance of a 6.7B-parameter GPT-J model, outperforming the much larger GPT-3 model across varied downstream tasks.
Takeaways & Limitations
The results show that self-supervised tool use can substantially improve zero-shot language-model performance across a range of downstream tasks.
Takeaways & Limitations
Toolformer cannot chain tools or interactively browse and refine searches because each API call is generated independently.
Abstract
from arXiv · showhide
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q\&A system, two different search engines, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.
1 Introduction
Toolformer addresses language models’ limitations by learning, without large-scale human annotation, when and how to use external APIs while preserving general language-modeling abilities. Built on pretrained GPT-J with 6.7B parameters, it achieves stronger zero-shot results across downstream tasks than larger GPT-3 and other baselines.
- Language models struggle to access up-to-date information and consequently tend to hallucinate.
- Toolformer learns to use external tools through self-supervision rather than large amounts of human annotations.The approach uses a handful of human-written API-use examples to generate candidate API calls and filters them according to their effect on language-modeling loss.
- Toolformer autonomously decides when and how to use which tool, enabling comprehensive tool use beyond task-specific settings.
- The dataset-agnostic approach uses the same pretraining dataset, preserving the model’s generality and language-modeling abilities.
- 6.7B parameters underlie Toolformer’s GPT-J base, which achieves stronger zero-shot results and clearly outperforms much larger GPT-3 and several other baselines.
2 Approach
Toolformer self-supervises tool use by representing API calls as text and augmenting plain-text data with calls whose results improve future-token prediction. It then fine-tunes the language model on this augmented data and executes APIs during decoding when the model signals that a response is expected.
- API representation: API calls are represented as text sequences with special tokens marking their boundaries, enabling insertion into arbitrary text.Each call is a tuple containing an API name and input, with its result represented in the linearized sequence.
- Data augmentation: The method samples candidate API calls from the model, executes them, filters them by usefulness for future-token prediction, and merges the survivors into the training data.Candidate positions are selected using a sampling threshold and capped at k positions; up to m calls are sampled per retained position.
- API filtering: An API call is retained only when providing both its input and result reduces weighted future-token loss by at least the filtering threshold τf.The comparison is against either making no API call or providing the call without its response.
- Model fine-tuning: The augmented dataset preserves the original texts while interleaving retained calls and results, and the model is fine-tuned with a standard language-modeling objective.Because calls are inserted where and with inputs that help prediction, fine-tuning teaches the model when and how to use tools from its own feedback.
- Inference: During inference, decoding pauses when the model produces the “→” token, after which the indicated API is called and its response is inserted before generation resumes.The response and the closing API token are inserted before decoding continues.
3 Tools
This section defines five text-in/text-out tools that address shortcomings of regular language models, requiring only a few demonstrations of intended use. The tools provide question answering, Wikipedia search, calculation, translation, and temporal context.
- Tool requirements: The tools accept text-sequence inputs and produce text-sequence outputs, with only a few demonstrations required for intended use.The five tools are question answering, Wikipedia search, calculator, calendar, and machine translation.
- Question Answering: Question answering uses Atlas, a retrieval-augmented language model fine-tuned on Natural Questions, to answer simple factoid questions.Atlas is based on another language model.
- Calculator: The calculator supports the four basic arithmetic operations and always rounds results to two decimal places.It performs simple numeric calculations.
- Wikipedia Search: Wikipedia search uses a BM25 retriever over the KILT Wikipedia dump, returning snippets that provide more comprehensive information but require the model to extract relevant parts.Unlike question answering, the search engine returns short text snippets for a search term.
- Machine Translation System: The translation system uses the 600M parameter NLLB model for 200 languages, automatically detects the source language, and always translates into English.The source language is detected with fastText, while English is fixed as the target.
- Calendar: The calendar API returns the current date without input, providing temporal context for predictions requiring awareness of time.It is queried without taking any input.
4 Experiments
Toolformer improves zero-shot performance across diverse tool-dependent tasks, often surpassing larger models, while preserving core language-modeling ability. Its benefits depend on task, language, model scale, and the tool’s usability.
- Zero-shot results: Toolformer improves over the best baseline by 11.7, 5.2, and 18.6 points across the evaluated tasks, also outperforming OPT (66B) and GPT-3 (175B).It independently asks the question-answering tool for required information in 98.1% of cases.
- Mathematical reasoning: Allowing API calls more than doubles Toolformer’s performance on every mathematical task, while Toolformer remains stronger than GPT-J and GPT-J + CC even without calls.The authors attribute the no-call improvement to finetuning on API calls and their results.
- Wikipedia search: Toolformer outperforms GPT-J-based baselines on the search benchmark but still trails GPT-3 (175B), relying on Wikipedia search in 99.3% of examples.The limitation is attributed to a simple search engine and Toolformer’s inability to reformulate queries or browse multiple results.
- Multilingual question answering: Machine-translation API calls consistently improve Toolformer across MLQA languages, with usage ranging from 63.8% to 94.9% except Hindi at 7.3%.Toolformer does not consistently beat vanilla GPT-J, partly because finetuning on CCNet can cause distribution-shift degradation.
- Temporal knowledge: Toolformer outperforms all baselines on TEMPLAMA and DATESET, but the responsible tools differ: calendar usage is 0.2% on TEMPLAMA and 54.8% on DATESET.TEMPLAMA gains mainly come from Wikipedia search and question answering, whereas DATESET gains are fully attributed to the calendar tool.
- Language modeling and scaling: Training on API-annotated data does not increase perplexity when calls are disabled, and effective tool use emerges at around 775M parameters.Smaller models show similar performance with and without tools, except for the comparatively easy Wikipedia search API.
5 Analysis
The analysis examines how decoding changes API-call frequency and performance, and finds that the filtering score generally distinguishes useful from uninformative calls. It also notes that perplexity with API calls enabled is intractable to evaluate directly.
- Limitations: Perplexity with API calls enabled is not evaluated because computing token probabilities would require intractable marginalization over all potential API calls.The limitation concerns pM(xt | x1, . . . , xt−1) when API calls are enabled.
- Decoding Strategy: Increasing k raises API-call rates from 40.3% on T-REx and 8.5% on WebQS at k = 1 to 98.1% and 100% at k = 10.The analysis compares decoding values using overall, API-call, and non-call subsets.
- Decoding Strategy: Greedy decoding already improves T-REx performance, whereas WebQS requires larger k before the model makes a substantial improvement.The passage attributes the difference to how decoding controls whether the model makes API calls.
- Data Quality: High L−i values typically correspond to intuitively useful API calls, while low values correspond to calls that provide no useful information for predicting future tokens.The score is used as a filtering criterion for API calls generated in CCNet text.
- Data Quality: Unfiltered noise can sometimes help by preventing the model finetuned on C∗ from blindly following every API result.The analysis gives “Fast train success” as an exception where an irrelevant call still reduces perplexity.
6 Related Work
Prior work augments language models with additional information or external tools, but typically provides information indiscriminately or relies on extensive data or task-specific prompting. Toolformer instead learns autonomously when and how to request useful information through self-supervision, extending related bootstrapping ideas.
- Language Model Pretraining: Pretraining methods add metadata, HTML tags, Wikipedia markup, or retrieved texts, but provide this information regardless of whether it helps prediction.Toolformer explicitly asks for the right information instead.
- Tool Use: Tool-use approaches commonly rely on large amounts of human supervision or few-shot prompts tailored to tasks with known tools.These approaches include search engines, web browsers, calculators, translation systems, and Python interpreters.
- Tool Use: Toolformer learns how and when to use tools without task-specific demonstrations, while TALM applies a similar self-supervised objective only to calculator and search-engine settings.TALM is identified as the approach most closely related to Toolformer.
- Bootstrapping: Bootstrapping methods have improved models across disambiguation, extraction, parsing, generation, classification, retrieval, and reasoning; Toolformer trains on its own predictions after perplexity-based filtering.This places Toolformer within a broader self-training and bootstrapping tradition.
7 Limitations
Toolformer has clear limitations in its current form, including an inability to chain tools by feeding one tool’s output into another tool’s input. This limitation arises because API calls are generated independently, leaving no chained-use examples in the fine-tuning dataset.
- Tool chaining: Toolformer cannot use tools in a chain, such as using one tool’s output as another tool’s input.The paper identifies this as a limitation of the current method.
- Tool chaining: Independent generation of each tool’s API calls prevents Toolformer from representing chained tool use.Each API call is generated independently rather than conditioned on another tool’s result.
- Tool chaining: Because the fine-tuning dataset contains no chained-tool examples, the current approach cannot learn this behavior.The absence of chained examples follows from generating API calls independently.
8 Conclusion
Toolformer learns self-supervised tool use through simple API calls, with sampled calls filtered by their effect on future-token perplexity. This substantially improves zero-shot performance for a 6.7B-parameter GPT-J model, which can outperform a much larger GPT-3 model across downstream tasks.
- Conclusion: Toolformer learns to use search engines, calculators, and translation systems through simple API calls in a self-supervised way.It is finetuned on sampled API calls.
- Conclusion: Sampled API calls are filtered according to whether they reduce perplexity on future tokens.This filtering criterion guides the finetuning data.
- Conclusion: Toolformer considerably improves zero-shot performance for a 6.7B-parameter GPT-J model across different downstream tasks.The conclusion reports broad zero-shot gains rather than a single task-specific result.
- Conclusion: The improved GPT-J model can outperform a much larger GPT-3 model on a range of downstream tasks.This comparison is made after incorporating Toolformer’s tool-use training.
A API Details
Toolformer filters and samples candidate API calls using probability and loss-reduction thresholds, while limiting the number of positions and calls considered per text.
- Sampling and filtering: API calls are considered where the <API> token probability is at least 5% and retained when they reduce loss by at least 1.0.The default thresholds are τs = 0.05 and τf = 1.0.
A.1 Implementation · A.2 Prompts
The implementation pairs specialized external tools with filtering and prompting procedures tailored to each tool. The prompts teach the model to insert API calls for answering questions, calculating, searching Wikipedia, translating, and using calendar information.
- A.1 Implementation: Atlas-large creates C∗ efficiently across millions of API calls, while Atlas-xxl answers questions during inference.The question-answering system is based on Atlas fine-tuned on Natural Questions.
- A.1 Implementation: The calculator uses a Python script supporting only +, −, ∗, and /, and returns no result for syntactically invalid equations.API-call sampling applies heuristic filters to CCNet based on numerical patterns and textual operation cues.
- A.1 Implementation: Calendar-date extraction assumes the document’s creation date and retains around 18% of documents after filtering out texts without extractable dates.The date is approximated by extracting it from the URL when present.
- A.1 Implementation: The machine-translation tool uses the 600M parameter NLLB model, detects source languages with fastText, and always translates into English.English-only text is filtered out before API-call generation, retaining non-English 10-token chunks surrounded by English text.
- A.2 Prompts: Question-answering prompts insert [QA(question)] calls for information needed to complete text, as illustrated by questions about entities and relationships.Examples ask where Joe Biden was born, what Coca-Cola is also called, and who manufactures it.
- A.2 Prompts: Wikipedia Search, machine-translation, and calendar prompts respectively insert [WikiSearch(term)], [MT(text)], and [Calendar()] calls where external information completes the text.Examples cover Ghana’s flag, nanomaterial risks, untranslated phrases, and calendar facts such as weekdays and days until Christmas.
B Toolformer Training · C Zero-Shot Prompts
Toolformer training uses a bounded, checkpoint-selected multi-GPU setup, while zero-shot evaluation applies task-specific prompts for factual completion, question answering, multilingual QA, and DATESET generation.
- B Toolformer Training: Training uses up to 25k examples per API with sequences capped at 1,024 tokens and an effective batch size of 128.Models are trained with DeepSpeed’s ZeRO-3 on 8 NVIDIA A100 40GB GPUs using BF16.
- B Toolformer Training: Models train for up to 2k steps, with perplexity evaluated every 500 steps on 1,000 CCNet development examples to select the best checkpoint.The selected checkpoint is the one that performs best on the small development set.
- C.1 LAMA and TEMPLAMA: For LAMA and TEMPLAMA, the prompt asks the model to complete input text so that it is factually correct.The input text x is inserted after the instruction and followed by a period.
- C.3 Question Answering: For question-answering datasets including DATESET, prompts prefix each question with “Answer the following question:” and add a question mark when needed.This provides a uniform question-answering instruction across the datasets.
- C.4 Multilingual Question Answering: For MLQA, the prompt asks the model to answer a question based on a paragraph and then answer in English.The context x and question q are inserted into the template.
- C.4 Multilingual Question Answering: DATESET templates randomly select a current_date and generate random past_date and future_date values for relevant fields.Holiday-related templates use federal holidays in the United States, including Thanksgiving.
C.2 Math Benchmarks · D DATESET
The math benchmarks use a fixed prompt format for answering questions from context, while DATESET tests date differences using randomly generated date pairs and calendar-based current-date information.
- C.2 Math Benchmarks: For all math benchmarks, the prompt presents context x and question q before “The answer is.”
- D DATESET: DATESET begins by randomly selecting 500 “current dates.”
- D DATESET: For each current date, DATESET randomly selects another past or future date within a four-year range.
- D DATESET: The two selected dates fill query templates listed in Table 11.
- D DATESET: One example asks, “How many days ago was August 14, 2020?”