PaperlayerView exact revision on arXiv

Source-linked AI summary

RestGPT: Connecting Large Language Models with Real-World RESTful APIs

Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, Sujian Li

arXiv:2306.06624v2cs.CL

TL;DR

Existing API-augmented LLMs are limited in their support for real-world APIs and complex instructions. RestGPT connects LLMs with RESTful APIs using coarse-to-fine online planning and a specialized executor, and experiments on RestBench report impressive results and strong robustness on complex tasks.

  • Problem

    Existing API-augmented LLMs mainly support specially designed tools and have difficulty fulfilling complex instructions in real-world scenarios.

  • Method

    RestGPT uses coarse-to-fine online planning for task decomposition and API selection, plus an executor that formulates RESTful API parameters and parses responses.

  • Results

    RestBench experiments on two realistic scenarios show that RestGPT achieves impressive results and strong robustness on complex tasks.

  • Takeaways & Limitations

    RestGPT demonstrates a framework for connecting LLMs with RESTful APIs and handling complex user instructions.

Abstract

from arXiv · show

Tool-augmented large language models (LLMs) have achieved remarkable progress in tackling a broad range of tasks. However, existing methods are mainly restricted to specifically designed tools and fail to fulfill complex instructions, having great limitations when confronted with real-world scenarios. In this paper, we explore a more realistic scenario by connecting LLMs with RESTful APIs, which adhere to the widely adopted REST software architectural style for web service development. To address the practical challenges of tackling complex instructions, we propose RestGPT, which exploits the power of LLMs and conducts a coarse-to-fine online planning mechanism to enhance the abilities of task decomposition and API selection. RestGPT also contains an API executor tailored for calling RESTful APIs, which can meticulously formulate parameters and parse API responses. To fully evaluate the performance of RestGPT, we propose RestBench, a high-quality benchmark which consists of two real-world scenarios and human-annotated instructions with gold solution paths. Experiments show that RestGPT is able to achieve impressive results in complex tasks and has strong robustness, which paves a new way towards AGI. RestGPT and RestBench is publicly available at https://restgpt.github.io/.

1 Introduction

Existing API-augmented LLMs remain limited in their support for real-world APIs and flexible planning. RestGPT addresses these challenges with coarse-to-fine online planning, RESTful API execution, and the RestBench benchmark.

  • 1 Introduction: Existing API-augmented LLMs support only small numbers of specially designed tools and struggle to adapt plans using API feedback.Prior approaches include offline plan-then-execute methods and ReAct, which face challenges generating viable adaptive plans.
  • 1 Introduction: RESTful APIs offer standardized endpoints, parameters, and response schemas through OpenAPI Specification, but their varied formats make invocation and parsing difficult.The RESTful setting is intended to improve extensibility while requiring robustness to unforeseen execution situations.
  • 1 Introduction: RestGPT uses iterative online planning: the planner generates sub-tasks, the API selector maps them to API plans, and the executor performs calls and parses results.The executor contains a caller for parameter formulation and a parser for API responses.
  • 1 Introduction: RestBench evaluates RestGPT on two realistic scenarios with human-annotated instructions requiring multiple APIs, and results show robust handling of complex instructions.The reported advantages span task planning, API understanding, and response parsing.
  • 1 Introduction: Connecting LLMs with RESTful APIs targets compatibility with existing real-world applications while providing extensibility.The contribution is framed as an initial attempt to connect LLMs with RESTful APIs.
  • 1 Introduction: RestGPT connects LLMs with RESTful APIs through a coarse-to-fine online planning framework addressing API understanding, planning, and response parsing.The framework uses a Planner, API Selector, and Executor.

2 Background

RESTful APIs provide a standardized, scalable interface for web-service functionality, while prior tool-augmented LLMs generally support fewer specially designed APIs and less flexible planning. RestGPT is presented as supporting over 100 RESTful APIs and using online planning to incorporate API feedback.

  • 2 Background: Prior tool-augmented LLMs are limited by support for specially designed APIs and planning methods that cannot interact with APIs to use feedback.The comparison identifies restricted API coverage and static offline planning as key limitations.
  • 2 Background: RestGPT stands out in the comparison by supporting over 100 RESTful APIs and using coarse-to-fine online planning rather than predominantly static offline planning.The cited comparison notes that API selection with retrieval is denoted by a dagger symbol.
  • 2 Background: RESTful APIs expose web-service functionality through standardized, stateless HTTP interfaces such as GET and POST.The background describes RESTful APIs as a common way to integrate external systems with client applications.
  • 2 Background: OpenAPI Specification documents RESTful endpoints, operations, parameters, and response schemas, providing a structured interface for API use.The specification is also known as Swagger.

3 RestGPT

RestGPT combines an LLM-based planner, API selector, and executor in an iterative coarse-to-fine loop. It uses execution feedback to revise plans and specialized caller and parser components to invoke RESTful APIs and extract useful results.

  • 3.1 RestGPT Architecture: RestGPT’s Planner, API Selector, and Executor decompose instructions, select APIs, perform RESTful calls, and extract useful information from JSON responses.Each component is implemented around an LLM prompt with in-context examples.
  • 3.1 RestGPT Architecture: The framework assigns different OAS portions to modules: the selector reads endpoint descriptions, while the caller uses detailed API documentation.This division addresses limited LLM context windows when handling many APIs.
  • 3.2 Coarse-to-Fine Online Planning: At each iteration, the planner generates a natural-language sub-task from the instruction, prior plans, and execution results, while the selector produces an API plan.The executor then returns an execution result for the next iteration.
  • 3.2 Coarse-to-Fine Online Planning: The planner can issue a continue signal when a sub-task is incomplete, prompting a new API plan based on the current plan and result.This mechanism enables online adjustment during execution.
  • 3.2 Coarse-to-Fine Online Planning: The planner issues an end signal after the user request is completed, allowing the framework to handle varied real-world execution situations.The continual and end states support flexible online planning.
  • 3.3 API Plan Execution: The executor’s caller generates API parameters or request bodies, while its response parser uses OAS schemas to generate Python code for extracting required information.API documents are filtered to preserve only APIs appearing in the current plan, and REST calls use the Requests library.

4 RestBench

RestBench evaluates complex multi-API instructions in two real-world scenarios using human-annotated gold solution paths. Its metrics assess successful completion, path correctness, and extra API calls beyond the gold path.

  • Benchmark scenarios: RestBench covers TMDB movie-database and Spotify music-player scenarios with instructions requiring multiple API calls.The benchmark targets both specialized information retrieval and control of a real-world application.
  • Instruction construction: Experts brainstormed diverse API combinations and annotated a gold API solution path for each instruction.Additional experts were used to support instruction quality, and solution paths provide a reference for evaluation.
  • Evaluation metrics: An API path is counted as correct when it contains the gold path as a subsequence, while human evaluation determines whether the result fulfills the request.The gold path elements need not be contiguous, and time-dependent requests prevent fixed ground-truth answers for every output.
  • Evaluation metrics: Table 4 reports success rate, Correct Path rate, and ∆Solution Length for the two RestBench scenarios.The table compares RestGPT with baselines and identifies the best results in boldface.
  • Evaluation metrics: ∆Solution Length measures the mean number of additional API calls required for successfully completed instructions.The metric compares actual and gold API-call counts over successful instructions.

5 Experiments

RestGPT outperforms the compared baselines on both RestBench scenarios and remains more effective as task complexity and API-set size increase. Error analysis and ablations identify planning, API selection, and response parsing as important sources of performance differences.

  • 5.2 Main Results: 75% success on the movie database and over 70% on the music player, outperforming all baselines while using the shortest solution length.Correct path rates are usually slightly higher than success rates because some plans fail during execution.
  • 5.3 Ablation Study: Removing the planner significantly degrades performance, while removing the schema-based parser also hurts performance on complex API responses.The results support the roles of coarse-to-fine planning and schema-based response parsing.
  • 5.2 Main Results: Most errors arise during planning, with planners exiting early and API selectors choosing incorrect APIs or hallucinating in-path parameters.ChatGPT makes more planning-stage errors than text-davinci-003, partly by continuing after instructions are fulfilled.
  • 5.4 Scaling Curves: When gold solution paths reach length 4, all baselines struggle, whereas RestGPT still achieves over 40% success.Success rates decline for all methods as instruction complexity increases.
  • 5.4 Scaling Curves: As noise APIs increase, baseline performance deteriorates while RestGPT remains almost unaffected.The test set contains 15 instructions solvable with 10 selected TMDB APIs before noise APIs are added.
  • 5.5 Case Study: In a case study, offline planning selects incorrect APIs and ignores dependencies, while ReAct generates difficult sub-tasks and consumes six API calls.RestGPT instead separates high-level sub-task generation from API selection and can issue a continuation signal when execution is incomplete.

6 Conclusion

The paper presents RestGPT for connecting LLMs to real-world applications through RESTful APIs. It combines online planning and specialized API execution with RestBench evaluation, reporting strong performance and robustness on complex tasks.

  • 6 Conclusion: RestGPT connects LLMs with real-world applications through RESTful APIs using coarse-to-fine online planning and a specialized executor.The executor formulates parameters and parses API responses.
  • 6 Conclusion: RestBench contains human-annotated instructions from two realistic scenarios for evaluating RestGPT.The paper reports impressive results on complex tasks and strong robustness.

A RESTful APIs and OAS

RESTful APIs expose networked resources through URLs and HTTP methods, while OAS standardizes how endpoints, parameters, requests, responses, and errors are described. These descriptions support API interaction and response parsing.

  • A RESTful APIs and OAS: RESTful APIs are a network-application architectural style in which resources are represented by URL endpoints accessed through HTTP methods.Examples of methods include GET and POST, and the paper cites services such as Spotify, Twitter, and Gmail.
  • A RESTful APIs and OAS: OAS is a JSON- or YAML-based specification that documents RESTful API endpoints, including operations, parameters, request and response formats, authentication, and errors.It standardizes how developers understand and interact with APIs.
  • A RESTful APIs and OAS: An OAS endpoint description can include a path, API description, request method, parameter list, response schema, optional response example, and error information.The response schema helps the parser extract useful information from JSON responses.

B Baselines

The baselines cover offline planning, interactive planning, interleaved reasoning and action, and reflective trial-and-error. RestGPT’s response parser addresses the complexity of RESTful API JSON responses by using OAS schemas to generate extraction code.

  • B Baselines: Offline introspective planning generates a multi-step API plan before execution and cannot adapt it to unforeseen situations without API feedback.DEPS instead iteratively updates plans using environment feedback until task completion.
  • B Baselines: ReAct interleaves chain-of-thought reasoning with actions, whereas Reflexion stores verbal reflections on feedback in episodic memory for later trials.These approaches represent interactive and reflective alternatives to offline planning.
  • B Baselines: Some baselines required reproduction and the proposed API executor so they could call RESTful APIs.This adaptation was used to compare methods originally designed for other settings.
  • B Baselines: RestGPT’s parser generates information-extraction code from OAS response schemas instead of directly prompting an LLM to read lengthy JSON responses.The paper illustrates the comparison with a 694-line RESTful API response.

D More Cases on RestBench

RestGPT is demonstrated on additional complex instructions across the TMDB and Spotify scenarios in RestBench.

  • RestBench includes two additional TMDB cases and two additional Spotify cases demonstrating RestGPT on complex instructions.Tables 5 and 6 cover TMDB, while Tables 7 and 8 cover Spotify.

E Failure Cases

The failure cases expose errors in API selection, planning, and response parsing that can produce incorrect answers.

  • The API selector hallucinated an in-path parameter, demonstrating a failure in selecting or formulating an API call.The first failure case involved inventing the tv_id parameter.
  • RestGPT returned an incorrect answer after redundant planning and incorrect response parsing caused it to extract an album name instead of a track.The executor had correctly provided the desired query result, but later pipeline errors led to the wrong final answer.

F Implementation Details

The implementation combines prompt-driven planning, API selection, calling, and response parsing in an iterative workflow, illustrated through TMDB examples and prompt specifications.

  • TMDB example: The TMDB example resolves Sofia Coppola’s person ID before selecting the movie-credits endpoint to count her directed movies.The example first selects GET /search/person, obtains ID 1769, and then selects GET /person/1769/movie_credits.
  • TMDB example: The caller executes documented REST requests and the parser extracts user-relevant information from JSON responses.The Sofia Coppola example returns a list of directed movies and concludes that she directed 14 movies.
  • TMDB example: A second TMDB example iteratively searches for Leonardo DiCaprio, identifies his latest movie, and retrieves its director.The final answer identifies Martin Scorsese as the director of Killers of the Flower Moon.
Loading 2306.06624v2…