Source-linked AI summary
Gorilla: Large Language Model Connected with Massive APIs
Shishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. Gonzalez
TL;DR
LLMs struggle to use APIs accurately because they must generate correct arguments and avoid hallucinating invalid calls while handling changing documentation. Gorilla combines self-instruct fine-tuning of a LLaMA-based model with document retrieval and evaluates it on APIBench, reporting stronger API-call performance than GPT-4 and reduced hallucination. The paper positions retrieval-aware training as a way to adapt API use to documentation changes, while noting that retrieval can sometimes hurt performance.
Problem
LLMs remain unreliable at API use because they may generate inaccurate arguments and hallucinate incorrect API calls despite broader reasoning and synthesis capabilities.
Method
Gorilla fine-tunes a LLaMA-7B model on self-instruct-generated instruction–API pairs, with document retrieval incorporated into training and inference.
Results
Gorilla surpasses GPT-4 in API functionality accuracy, reduces hallucination errors, and adapts to changes in API documentation when retrieval is used.
Takeaways & Limitations
APIBench and Gorilla provide a framework for more accurate API-tool use across large, changing collections of machine-learning APIs.
Takeaways & Limitations
Retrieval does not always improve performance and can sometimes hurt it; the study also focuses on machine-learning APIs that may produce biased predictions from skewed training data.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, enabling flexible user updates or version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, we introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep up with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. Gorilla's code, model, data, and demo are available at https://gorilla.cs.berkeley.edu
1 Introduction
LLMs can reason and synthesize programs but remain constrained by fixed knowledge, static computation, and limited context, making large collections of changing APIs difficult to use. Gorilla addresses this challenge with self-instruct fine-tuning and retrieval, evaluated on APIBench, and reports improved API accuracy with fewer hallucinations.
- LLMs remain limited by fixed stored information, static computation, and limited context, requiring retraining as the world changes.
- Tool access can expand LLMs’ knowledge and computational capabilities, but prior integrations generally used small, well-documented API sets.
- Web-scale API usage requires retrieval because millions of changing, overlapping APIs cannot fit in one context and have nuanced constraints.
- APIBench combines TorchHub, TensorHub, and HuggingFace APIs with synthetic instruction prompts to evaluate API selection and functional correctness.The dataset includes 94 TorchHub APIs, 696 TensorHub APIs, and 925 HuggingFace models selected by download frequency.
- Gorilla is a fine-tuned LLaMA-based model using document retrieval that outperforms GPT-4 on API functionality accuracy, reduces hallucinations, adapts to documentation changes, and reasons about constraints.
2 Related Work
Prior work explored tool use, prompting, and program synthesis, but generally focused on specific tools or general coding rather than systematic API evaluation across a broad tool space. Gorilla instead targets linear API-call programs, which are easier to verify while expanding access to tools without requiring low-level implementation.
- Tool Usage: Tool-use research has examined web browsing, calculators, translation, and Python interpreters, but generally concentrates on specific tools.
- Tool Usage: Gorilla explores a vast, open-ended array of API tools rather than a small collection of predefined tools.
- Tool Usage: Prior API-call work primarily demonstrates prompting, whereas Gorilla emphasizes systematic evaluation and a training pipeline for API use.
- LLMs for Program Synthesis: Program-synthesis research uses prompting, decomposition, self-debugging, and code pretraining to improve general code generation.
- LLMs for Program Synthesis: Gorilla focuses on linear programs composed of API calls, a restricted domain that functions more like tool use and avoids low-level implementation details.
3 Methodology
The methodology builds APIBench from model-hub documentation, trains Gorilla with self-instruct data and optional retrieval, and evaluates API selection through constraint reasoning and AST subtree matching.
- Dataset Collection: APIBench aggregates model-card APIs from HuggingFace, TorchHub, and TensorHub, with each API represented by structured metadata such as arguments, requirements, examples, and descriptions.The collection records online model cards and converts API calls into JSON objects designed to generalize beyond machine-learning APIs.
- Instruction Generation: GPT-4 generates synthetic instruction–API pairs from API documentation under instructions that hide API names and usage hints.The self-instruct process uses in-context examples and samples multiple instructions for each API datapoint.
- Gorilla: Gorilla is trained by instruction-finetuning LLaMA-7B on user–agent conversations, with experiments conducted both with and without retrieved API documentation.Retriever-aware training appends retrieved API documentation to the user prompt so the model learns to use documentation when answering the request.
- API Call with Constraints: API constraints require Gorilla to jointly interpret an API’s functionality and parameters such as model size or accuracy thresholds.The paper frames constrained image-classification requests as requiring reasoning over multiple requirements rather than functionality alone.
- Gorilla Inference: During inference, Gorilla accepts natural-language requests in zero-shot or retrieval mode and returns an API call for the requested task.Retrieval supplies API documentation alongside the user prompt, while zero-shot inference uses the prompt alone.
- Verifying APIs: AST subtree matching verifies whether a generated single API call corresponds to a dataset API by matching required function and argument nodes while ignoring optional defaults.The method identifies hallucinations when a generated call is not a subtree of any database API; optional arguments such as pretrained=True are excluded from matching.
4 Evaluation
Evaluation compares Gorilla with state-of-the-art models across API-call accuracy, retrieval settings, documentation changes, hallucination, and constraint-aware invocation. Gorilla leads zero-shot accuracy, benefits from retrieval-aware fine-tuning, adapts to API changes, and navigates constraint trade-offs.
- Evaluation setup: Gorilla is benchmarked against state-of-the-art language models using zero-shot and BM25, GPT-Index, and Oracle retrieval settings.The evaluation covers Torch Hub, HuggingFace, and TensorFlow Hub APIs.
- AST accuracy: 20.43% better than GPT-4 and 10.75% better than ChatGPT, Gorilla achieves state-of-the-art zero-shot performance across the evaluated models.The reported improvement over LLaMA reaches 83%.
- Retrieval ablations: 21.50% lower in Torch Hub and 47.57% lower in HuggingFace, non-optimal BM25 or GPT-Index retrieval can reduce performance when Gorilla was not fine-tuned with retrieval.Ground-truth retrieval changed performance by 0.88% worse in TensorHub and 0.97% better in HuggingFace.
- Retrieval-aware fine-tuning: 12.37% better in Torch Hub and 23.46% better in HuggingFace, retrieval-aware fine-tuning outperforms training without retrieval under ground-truth retrieval.At evaluation time, GPT-Index causes 29.20% accuracy degradation and BM25 causes 52.27% degradation.
- API Call with Constraints: With constraints, accuracy drops across models, while retrieval-enabled Gorilla matches GPT-3.5’s best performance and achieves the highest zero-shot accuracy.The constrained evaluation concerns Torch Hub models with accuracy defined for at least one dataset.
5 Conclusion
Gorilla is a fine-tuned LLaMA-based pipeline for API calling that outperforms GPT-4 across three API datasets. It reduces hallucinations, adapts to changing documentation, and handles API constraints.
- Gorilla is a LLaMA-based model and pipeline fine-tuned to call APIs across three massive datasets.
- Gorilla surpasses GPT-4 in API functionality accuracy while reducing hallucination errors.
- Retriever integration enables Gorilla to adapt to test-time API documentation changes.
- Gorilla generates reliable API calls without hallucination and can satisfy constraints when selecting APIs.
6 Limitations & Social Impacts
The dataset focuses on ML APIs, whose predictions can reflect bias from skewed training data. The authors release over 11,000 instruction–API pairs to support broader understanding of these APIs.
- ML APIs were chosen to create a challenging dataset because of their functional similarity.
- Skewed training data can make ML APIs produce biased predictions that disadvantage certain sub-groups.
- The released resource contains over 11,000 instruction–API pairs for the wider community.
8 Appendix
The appendix details APIBench’s domains, instruction design, evaluation setup, training procedure, and model comparisons. It reports strong zero-shot performance for Gorilla and reduced hallucination relative to GPT-4, especially across structured API calls.
- Dataset: Torch Hub, Tensor Hub, and HuggingFace contain 95, 696, and 925 APIs, respectively.
- Dataset: Each API receives 10 unique instructions designed for training and evaluation.
- Evaluation: The API Call Task evaluates single-line code generation either zero-shot or with an API reference.
- Limitations: Code examples are provided but de-prioritized because their execution results were not tested.
- Training: Gorilla is trained for 5 epochs with a 2e-5 learning rate using cosine decay and 8×A100 GPUs with 40G memory each.
- Results: Gorilla surpasses GPT-4 and GPT-3.5 by a large margin in zero-shot settings, while those baselines reach around 40% accuracy on Torch Hub and Tensor Hub.
- Evaluation: The dataset uses domain-specific API and argument checks, including required model identifiers or handles for supported calls.
- Results: GPT-4 shows serious HuggingFace hallucinations involving invalid repository names or assumed local paths, which Gorilla greatly reduces.