PaperlayerView exact revision on arXiv

Source-linked AI summary

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, Niranjan Balasubramanian

arXiv:2407.18901v1cs.SEcs.AIcs.CLcs.LG

TL;DR

Everyday digital tasks span apps and require interactive reasoning, sequential decisions, and handling unexpected hurdles, while existing benchmarks lack suitable complexity and interactivity. AppWorld combines a controllable, reproducible execution environment with a benchmark of complex API-based tasks and programmatic evaluation. ReAct + GPT4O scores 48.8 TGC on Test-N and 30.2 TGC on Test-C, and evaluated methods show low task and scenario completion across both sets.

  • Problem

    Everyday digital tasks span apps and require interactive reasoning, sequential decisions, handling unexpected hurdles, and avoiding undesirable outcomes, while benchmarks lack suitable complexity and interactivity.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Method

    AppWorld combines a controllable, reproducible execution environment with a benchmark of complex API-based everyday tasks and programmatic evaluation.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    ReAct + GPT4O achieves TGC 48.8 on Test-N and 30.2 on Test-C, while all evaluated methods score low on task and scenario completion across both sets.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    AppWorld's evaluation underscores the difficulty of automating everyday digital tasks for modern LLMs and highlights open challenges.

    Source: §6 · p. 9

    AppWorld benchmarks API-based tasks rather than UI modalities, though future work could add a UI over its API backend.

    Source: §7 · p. 10

Abstract

from arXiv · show

Autonomous agents that address day-to-day digital tasks (e.g., ordering groceries for a household), must not only operate multiple apps (e.g., notes, messaging, shopping app) via APIs, but also generate rich code with complex control flow in an iterative manner based on their interaction with the environment. However, existing benchmarks for tool use are inadequate, as they only cover tasks that require a simple sequence of API calls. To remedy this gap, we built $\textbf{AppWorld Engine}$, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users. We then created $\textbf{AppWorld Benchmark}$ (40K lines of code), a suite of 750 natural, diverse, and challenging autonomous agent tasks requiring rich and interactive code generation. It supports robust programmatic evaluation with state-based unit tests, allowing for different ways of completing a task while also checking for unexpected changes, i.e., collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our 'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least 16% fewer. This highlights the benchmark's difficulty and AppWorld's potential to push the frontiers of interactive coding agents. The project website is available at https://appworld.dev/.

1 Introduction

AppWorld targets day-to-day automation that requires operating multiple apps, writing code iteratively, and reasoning through interactions—capabilities existing tool-use benchmarks inadequately test. It combines a controllable app simulator, complex tasks, and programmatic state-based evaluation.

1 Introduction

  • Everyday automation spans apps and requires interactive reasoning, sequential decisions, hurdle handling, and avoiding undesirable outcomes, while prior benchmarks use linear sequences of 1–4 API calls.The introduction also notes that reference-solution comparisons are unsuitable for complex tasks with many valid solutions.Source: §1 · pp. 1–2
  • AppWorld Engine provides a stable, controllable simulator of 9 everyday apps, populated with activities of about 100 fictitious users and containing 101 database tables and about 457 APIs.Agents operate through APIs without real-world consequences or resource use.Source: §1 · p. 2
  • AppWorld Benchmark offers 750 tasks requiring iterative code generation, with averages of 1.8 apps, 9.5 APIs, and 50 code lines per task.Tasks reach maxima of 6 apps, 26 APIs, and 134 code lines, and are designed to require planning, adaptation, and handling failures.Source: §1 · p. 2
  • Its evaluation suite uses an average of 8 database-checking unit tests per task to assess goal completion while allowing different valid solutions.Tasks have up to 22 tests, which directly check database entries and changes.Source: §1 · p. 2
  • GPT4O with ReAct achieves 48.8 on normal and 30.2 on challenge task goal completion, versus GPT4Trb’s 32.7 and 17.5; its scenario-based scores fall to 32.1 and 13.0.The scores are reported for the normal and challenge test sets, respectively.Source: §1 · p. 2

2 AppWorld Engine

AppWorld Engine is a controllable simulator of day-to-day apps and people, designed for agents to operate through APIs without real-world consequences. It combines documented app interfaces and a stateful execution shell with realistic, reproducible app behavior and a populated database.

2.1 Engine Components

  • The engine implements 9 day-to-day apps with 457 APIs and 1,470 arguments, plus ApiDocs for interactive documentation lookup and Supervisor APIs for task-assigner information.API calls simulate app activities through database reads and writes.Source: §2.1 · p. 3
  • Each API documents its purpose, parameter types and constraints, and response JSON schema; agents use a stateful Jupyter-style shell with informative errors and direct or REST API calls.The shell supports safe code execution, reusable variables, serverless API invocations, and HTTP requests in any programming language.Source: §2.1 · p. 3

2.2 Features

  • AppWorld simulates realistic app behavior, including consistent within- and cross-app state changes, authentication and access controls, and paginated search; database and time control enable reproducible starting states.Starting states can be customized for tasks and used to support programmatic evaluation.Source: §2.2 · pp. 3–4
  • The team reports 1,780 unit tests across 34K lines of test code and 98% code coverage, and reviews documentation for accuracy alongside parts generated from API code.The testing is intended to ensure APIs respond appropriately to varied agent behavior.Source: §2.2 · p. 4

2.3 Database

  • The Base DB contains 101 tables and 726 columns and about 370K rows; it models 106 fictitious people and their app activities, using procedural population to sample realistic, diverse data at scale.Source: §2.3 · p. 4

3 AppWorld Benchmark

AppWorld Benchmark uses scenario-based task generators to create varied instructions and starting states, then pairs tasks with state-based evaluation and programmatic solvability checks.

3.1 Setup

  • Scenario generators vary placeholder values and starting states to create multiple tasks from each natural-language task scenario.Setup tailors each task to be well-defined, include distractors and realistic hurdles, and form a contrast set with other tasks from its scenario.Source: §3 · p. 4, §3.1 · p. 5

3.2 Evaluation

  • State-based evaluation checks final database changes against required and allowed changes, supporting valid solution paths while detecting collateral damage.The evaluator uses programmatic assertions rather than comparing an agent’s steps with a reference solution.Source: §3.2 · pp. 5–6
  • Task Goal Completion (TGC) measures the percentage of tasks passing all evaluation tests, while Scenario Goal Completion (SGC) measures the percentage of scenarios whose tasks all pass.SGC assesses reliability across varying requirements and starting states within a scenario.Source: §3.2 · p. 6

3.3 Validation Solutions

  • Fully programmatic solutions run from each task’s starting database to verify that tasks are solvable under their evaluation suites.The authors also use these end-to-end tests to check that tasks remain solvable after future changes to apps, APIs, or the base database.Source: §3.3 · p. 6

3.4 Final Dataset

  • The benchmark has 250 scenarios with three tasks each (750 total), split into Train (105), Dev (60), Test-N (168), and Test-C (417).Test-C tasks require at least one designated unseen app, Amazon or Gmail; they tend to be harder and require more solution-code lines than Test-N, and are the benchmark’s main target.Source: §3.4 · pp. 6–7
  • Across scenarios, 86% of tasks require a unique API set spanning 52 app combinations, and the benchmark includes three subjective difficulty levels and tasks requiring rich code.Tasks also include hurdles and distractors; validation solutions demonstrate the programming constructs, code length, and API use required.Source: §3.4 · p. 7

4 Benchmarking Experiments

The experiments evaluate iterative agents on AppWorld and find low task and scenario completion, with performance worsening as task difficulty rises. Oracle API results suggest that API retrieval is not the main bottleneck; agents also struggle to use APIs in complex code and adapt to feedback.

4.1 Methods

  • The study evaluates iterative decision-making and code-generation approaches, using an API predictor to supply relevant documentation because the full API documentation spans ~142K tokens.The evaluated base models include GPT4O, GPT4Trb, LLaMA3, and DeepSeek; CodeAct and ToolLLaMA use specialized base models.Source: §4.1 · pp. 7–8

4.2 Results

  • 48.8 TGC on Test-N and 30.2 TGC on Test-C for ReAct + GPT4O; the best open model, FullCodeRefl + LLaMA3, achieves 24.4 and 7.0 TGC, respectively.Across methods, scenario completion scores are 30–50% below task completion scores, indicating inconsistent completion of task variants within a scenario.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 Analysis

  • ReAct + GPT4O TGC falls from 58.3 at difficulty level 1 to 21.0 at level 3, and tasks requiring 60+ lines of code have TGC < 20.Performance also worsens as the number of unique APIs increases.Source: §4.3 · p. 8, Figure 4 · p. 8
  • Oracle APIs improve TGC by up to 7.8 points on Test-N and 9.8 on Test-C, but the best Oracle API TGC scores are only 54.8 and 35.2, respectively.The authors conclude that API retrieval is not the main bottleneck; agents struggle to use APIs in complex interactive code and adapt to errors and other messages.Source: §4.3 · p. 8
  • Prompt variations improve Test-N by up to 6.6 TGC and 5.3 SGC but do not improve Test-C or exceed the best 48.8 TGC.The variations include more demonstrations, selecting examples by predicted API similarity, and reordering demonstrations where applicable.Source: §4.3 · p. 9
  • Manual error analysis identifies failures in interaction, API understanding and usage, instruction following, and commonsense, as well as repeated work that exhausts LLM-call budgets.Source: §4.3 · p. 9

5 Related Work

AppWorld combines complex executable APIs with interactive code generation for daily-life tasks and evaluates task completion programmatically rather than by matching a single solution path.

Tool-Usage Benchmarks

  • Prior tool-use benchmarks require only 1-4 API calls, whereas AppWorld tasks require up to 26 APIs (avg. 9.5) and embed calls in programs with up to 134 (avg. 41.3) lines of code.AppWorld tasks also involve dependencies between calls and environment interaction.Source: §5 · p. 9
  • AppWorld uses process-agnostic programmatic criteria to check task completion, unlike prior evaluation methods that compare tool calls and answers with a gold solution or rely on LLMs or human judgment.This supports evaluation when tasks have multiple valid solution paths.Source: §5 · p. 9

(Interactive) Code Generation Benchmarks

  • Existing code-generation benchmarks do not combine interactive code generation with a large range of complex APIs for daily-life tasks.Prior work spans short code snippets, patch generation, interactive code execution, and Python-based tool use, but Tool3Eval is limited to narrow domains and a few tools.Source: §5 · p. 9

6 Conclusions

AppWorld provides a controllable, reproducible environment and interactive coding benchmark for complex everyday API tasks, with programmatic evaluation to assess autonomous agents. Its modular Engine also supports extensions for studying broader forms of digital-task automation.

6 Conclusions

  • AppWorld introduces a controllable, reproducible execution environment and an interactive coding benchmark for complex everyday API-based tasks.The framework addresses the lack of a unified, stable executable environment and suitably complex, interactive benchmarks.Source: §6 · p. 9
  • Its programmatic evaluation suite and realistic hurdles and distractors enable rigorous assessment, while model benchmarking underscores AppWorld's difficulty and open challenges for LLMs.The open challenge is automating everyday digital tasks.Source: §6 · p. 9
  • AppWorld Engine is modular and extensible, enabling possible extensions for UI control, multi-agent or human collaboration, and study of privacy and safety risks.The risks concern digital assistants acting on users' behalf in the real world.Source: §6 · p. 9

7 Limitations

AppWorld’s limitations are its API-only interaction scope, focus on single-assistant tasks, and benchmark size, which is insufficient for model training. The authors identify UI support, multi-agent tasks, and LLM-based data augmentation as directions for future work.

7 Limitations

  • AppWorld exposes app functionality through APIs, not UI modalities such as browsers or mobile interfaces; a UI layer could extend tasks while preserving their state-based evaluation.The tasks and evaluators are agnostic to the process used to achieve the goal.Source: §7 · p. 10
  • The benchmark covers autonomous single-assistant tasks, not multi-agent coordination, although AppWorld can already simulate such scenarios for future benchmark extensions.An example is scheduling a meeting between two supervisors’ assistants via email.Source: §7 · p. 10
  • The carefully crafted benchmark has too few instances for training models; bootstrapping or self-instruction with LLMs could provide cheaper, noisier augmentation.The authors identify these as potential future techniques for data augmentation.Source: §7 · p. 10

8 Ethics Statement · A Strong Interaction Requirement

AppWorld’s tasks often require agents to interact with the environment to interpret information, while the paper notes ethical limitations and uses a sandbox and side-effect checks to support safe development.

8 Ethics Statement

  • Even without sensitive attributes in its tasks, AppWorld models may exhibit social, cultural, and ethical biases inherited from their underlying LLMs.Source: §8 · p. 10
  • Authors at North American and European institutions designed the benchmark tasks, which may not reflect digital lives elsewhere, although automating such tasks remains valuable for research and practice.Source: §8 · p. 10
  • Real-world automation poses risks, but AppWorld results remain far from deployment; its sandbox enables experimentation and its evaluation checks for unintended side effects.The paper also notes the potential efficiency benefits of such assistants.Source: §8 · p. 10
  • ChatGPT-assisted data population was reviewed by a human, but missed personal information or offensive content remains possible; the benchmark uses a permissive license and original app API implementations.The paper says the APIs do not infringe the corresponding real apps’ copyrights and that the tools used to build AppWorld are open-source and free to use.Source: §8 · p. 10

A Strong Interaction Requirement

  • Most AppWorld tasks, especially harder ones, require environment interaction to interpret free-form information; weaker tasks can be solved in one shot with perfect API knowledge.The paper contrasts finding a workout playlist from free-form notes with playing a playlist identified by title, and says agents still benefit from exploration when API knowledge is imperfect.Source: A Strong Interaction Requirement · p. 13
  • Removing response schemas can make weak-interaction tasks require strong interaction, since agents must inspect outputs; future work could address missing, incomplete, or outdated API documentation.The paper notes that many real-world API documentations have poorly maintained response schemas.Source: A Strong Interaction Requirement · p. 13

B AppWorld Benchmark Distributions · C AppWorld Execution Shell

The benchmark section characterizes task distributions, while the execution shell offers interactive code execution and app access alongside convenience and safety features. Its core requirement remains making HTTP calls, independent of language or process.

B AppWorld Benchmark Distributions

C AppWorld Execution Shell

  • The IPython-based shell supports stateful execution, letting agents inspect outputs and reuse variables across code blocks.It enables iterative interaction in a manner similar to a Jupyter Notebook.Source: C AppWorld Execution Shell · p. 13
  • Agents can call app APIs through direct Python functions or REST requests; experiments use direct calls for simplicity, though both are supported.REST calls require selecting an HTTP method and constructing the URL and parameters from documentation.Source: C AppWorld Execution Shell · p. 13
  • The shell provides best-effort restrictions on Python access to system-level modules, with a Docker image using gVisor for containerization.The passage describes disabling access to most, if not all, modules affecting the local file system or process.Source: C AppWorld Execution Shell · p. 13
  • Informative errors cover Python failures, failed API requests, infinite loops, and execution time limits.API requests are treated as failed when their status code is not 200 or 201.Source: C AppWorld Execution Shell · p. 13
  • The shell freezes date and time to each task’s configured values, so calls such as datetime.now() return task-specific time.The shell uses the freezegun library.Source: C AppWorld Execution Shell · p. 13
  • A single Python process combines the SQLite-backed database and FastAPI Test Client, avoiding process management and loading tasks in under 0.5 seconds on average.The reported average is on a regular MacBook Pro; the framework avoids restarting processes between tasks despite their different databases and states.Source: C AppWorld Execution Shell · pp. 13–14
  • Shell features are quality-of-life conveniences rather than requirements: agents need only make HTTP calls, and state-based evaluation is process-agnostic.The HTTP calls can be made in any programming language.Source: C AppWorld Execution Shell · p. 14

D Hash-based Database Difference … E.7 Other Details

The appendix details a hash-based method for fast database-state diffs and describes the modeling approaches and evaluation settings used for the agent baselines.

D Hash-based Database Difference

  • Evaluations run in under 0.6 seconds per task on average, with hierarchical hashes enabling fast table-, row-, and column-level diffs.Row hashes exclude IDs and change when row content changes; table counters help identify candidate tables, while unchanged tables skip row- and column-level comparisons.Source: D Hash-based Database Difference · pp. 14–15

E Modeling Details; E.1 FullCodeRefl

  • FullCodeRefl generates code in one pass, then uses execution stack traces for natural-language reflection and retries, capped at five.Prompts for the methods are in Appendix K, and Figure 8 depicts the first three methods.Source: §E.1 · p. 15, E Modeling Details · p. 15, Figure 8 · p. 16

E.2 ReAct

  • ReAct alternates reasoning and code execution until task completion or 100 LLM calls, using error traces to guide corrections.It explores API documentation through the ApiDocs app; when trajectories exceed 20,000 characters, it removes older observations first, then older interaction blocks.Source: §E.2 · pp. 15–16

E.3 PlanExec

  • PlanExec first generates a plan, then executes each step in ReAct-style sequences, with prior step history available to the executor.The planner is instructed to make each step independently executable; execution uses ReAct's truncation strategy and explores API documentation as needed.Source: §E.3 · p. 16, Figure 8 · p. 16

E.4 IPFunCall

  • IPFunCall batches independent API calls, feeds their collective outputs back iteratively, and uses a 15-turn limit where Dev-set performance saturated.Each turn can contain any number of function calls, allowing a batch to act effectively like a program loop.Source: §E.4 · pp. 16–17

E.5 ToolLLaMA

  • ToolLLaMA generates sequences of API calls and was tested with its official codebase, but the reported setup did not solve any tasks.The authors describe it as a multi-turn-trained tool-use model and also tried minimal prompt changes and few-shot demonstrations adapted from IPFunCall.Source: §E.5 · p. 17

E.6 CodeAct

  • CodeAct uses Mistral-7B trained on multi-turn coding and self-debugging trajectories, but the tested setup solved no tasks.The authors changed its stopping rule to task completion or a 10-interaction maximum rather than oracle evaluation.Source: §E.6 · p. 17

E.7 Other Details

  • The experiments used GPT-4, GPT-4o, GPT-4 Turbo, LLaMA 3, and DeepSeek, with greedy decoding at temperature 0 and top-p 1.0.Source: §E.7 · p. 17

E.8 Cost of Experiments

Under June 2024 OpenAI pricing, the experiments cost around $10K, limiting further exploration in the modeling space; the authors call for improving agent efficiency alongside completion rates.

E.8 Cost of Experiments

  • At $5 per 1M tokens (June 2024 rates), GPT4O cost $0.7 per example for ReAct, $1.33 for PlanExec, $0.33 for IPFunCall, and $0.02 for FullCodeRefl on Test-N.Source: §E.8 · p. 17
  • GPT4Trb cost about twice as much as GPT4O and GPT4 six times as much; across experiments, total costs were around $10K, limiting further modeling exploration.Source: §E.8 · p. 17
  • Future work should improve agent efficiency as well as completion rates, including by reducing cost, time taken, or LLM-token use.Source: §E.8 · p. 17

F Additional Experiments … F.5 Additional Base LLM explored

Additional experiments find that performance varies with task difficulty and base LLM, while more or reordered demonstrations do not consistently improve results. API retrieval is generally strong on Test-N, but retrieval challenges are not the primary difficulty on Test-C.

F.1 API Predictor performance

  • API retrieval scores are generally high for Test-N, with room to improve especially on Test-C; retrieval challenges are not the benchmark’s primary difficulty.The paper attributes the latter conclusion to its discussion in §4.3.Source: §F.1 · p. 17

F.2 Demonstration Selection

  • The demonstration-selection exploration was limited to FullCodeRefl because its validation solutions could be automatically reused as demonstrations; other methods required per-example annotation.Source: §F.2 · p. 18
  • For FullCodeRefl, additional demonstrations improve GPT4O scores on Test-N, but not Test-C, and performance still lags far behind ReAct.The exploration used random or nearest examples; nearest examples were selected by overlap between predicted APIs and gold APIs.Source: §F.2 · p. 18, Table 5 · p. 18

F.3 Demonstration Reordering

  • Changing the order of GPT4O’s three demonstrations produced some variation, but no scores surpassed the best reported in Table 3.ReAct was omitted because its long trajectories allow only one demonstration and thus only one permutation.Source: §F.3 · p. 18, Table 6 · p. 18

F.4 Scores across Hardness Predictors

  • Across methods, GPT4O performs worse as hand-labeled task difficulty, lines of code, and unique APIs increase.The reported trend is similar across the task-difficulty indicators.Source: §F.4 · p. 18

F.5 Additional Base LLM explored

  • On Test-N, Gemini Pro 1.5 scored 20.2 TGC and 10.7 SGC, below LLaMA3’s 24.4 TGC and 17.9 SGC; GPT4 was much worse and 6X more expensive than GPT4O.The authors therefore did not test GPT4 on the larger Test-C set or explore Gemini Pro 1.5 further, citing early performance indications and restrictive per-day API rate limits.Source: §F.5 · p. 18, Table 7 · p. 18

G Comparison with Web and UI Navigation Benchmarks

Web and UI navigation benchmarks cover text- or visually represented interfaces, while AppWorld’s interactive coding uses APIs; the paper also reports declining performance as task difficulty rises.

G Comparison with Web and UI Navigation Benchmarks

  • WebArena (Zhou et al., 2024) and related benchmarks use textual website HTML, while WebVoyager (He et al., 2024) and others add visual representations.The passage also names Mind2Web and WebShop among textual-representation benchmarks, and VisualWebarena, WebLINX, and MMInA among visually enabled benchmarks.Source: G Comparison with Web and UI… · p. 18
  • Figure 9 shows methods perform worse as task difficulty increases across Test-C and Test-N hardness indicators and GPT4O scores.The figure presents TGC and SGC scores alongside hardness indicators.Source: Figure 9 · p. 19
  • Mobile UI benchmarks target phone apps and OSWorld desktop apps; UI navigation selects screen elements, whereas interactive coding composes documented API calls.The passage describes UI interaction through actions such as clicking, typing, and scrolling, in contrast to writing code based on API documentation and execution outputs.Source: G Comparison with Web and UI… · p. 19

H AppWorld Benchmark Examples · I Example API Unit Tests

The section illustrates AppWorld Benchmark scenarios and task instructions across hand-labeled difficulty levels, then reports the Engine’s API unit-test validation and coverage.

H AppWorld Benchmark Examples

I Example API Unit Tests

J Task Generator Example · K Prompts · L App Documentation

This section illustrates AppWorld task construction and evaluation through a Spotify example, presents prompts for API selection and iterative Python-based task completion, and describes the API documentation available to agents, including supervisor endpoints.

J Task Generator Example

  • Setup programs ensure scenario tasks are solvable, include distractors and hurdles, and form contrast sets; the example task links a Spotify workout playlist to a plan in Simple Note.The example setup selects a workout day and duration and prepares playlists around that duration.Source: Table 8 · p. 20, Table 9 · p. 21, J Task Generator Example · p. 23
  • The gold solution finds the workout plan and plays a sufficiently long playlist; evaluation checks task completion, playback, queue duration, and the exact queued song IDs.The evaluation compares the predicted answer with the ground truth and verifies the music player’s end state.Source: J Task Generator Example · pp. 25–27

K Prompts

  • The authors ask readers not to reveal benchmark solutions online, except those shown in the paper, to reduce leakage into foundation-model training corpora.They also include a canary string in the code and data to help training corpora filter it out.Source: K Prompts · p. 27
  • The API predictor prompt favors recall by asking for all potentially useful APIs while limiting output to 16 entries.It specifies one API per line in app_name.api_name format.Source: K Prompts · p. 27
  • FullCodeRefl asks for valid Python using documented, allowed APIs, requires task completion to be signaled, and permits reflection and retries after errors.The retry prompt says earlier database changes persist, so retries must account for effects of previous attempts.Source: K Prompts · pp. 28–29
  • ReAct uses a multi-step Python REPL, where agents inspect app, API, and API-specification documentation, execute code, and use outputs to choose subsequent steps.Its example pages through playlist results before returning the count and completing the task.Source: K Prompts · pp. 30, 32
  • PlanExec separates planning from API execution: the planner receives app descriptions and must provide clear, independently executable steps with all relevant variables.The executor model handles API interactions.Source: K Prompts · p. 33
  • Planning instructions favor enough steps to complete a task and require the plan to end with either its requested answer or a completion step.The prompts distinguish reporting an answer from tasks that require no answer, and reserve the complete_task call for instructions that explicitly require it.Source: K Prompts · pp. 33, 37, 38
  • Prompt examples decompose Spotify lookup, cross-app messaging, and Venmo calculations into ordered steps that end with the requested answer or task completion.The examples include retrieving song likes, collecting movie titles from notes and texting them, and summing payments to roommates.Source: K Prompts · pp. 33–34
  • The executor prompt has agents write and run Python code in a multi-step REPL conversation, using each execution's output to inform subsequent API calls.It also introduces APIs for discovering available apps, listing an app's APIs, and inspecting an API's specification.Source: K Prompts · p. 34
  • The prompts direct agents to consult API specifications, seek another API when returned data lacks needed metadata, and loop through pages to retrieve all items.They also recommend writing small code chunks and checking that they work before making irreversible changes.Source: K Prompts · p. 38
  • Agents interact with the execution environment over multiple turns, may make multiple function calls per turn, and must finish within 15 turns.The prompt advises combining requests where possible and ending by calling the Supervisor completion API.Source: K Prompts · p. 38
  • The CodeAct prompt exposes app APIs as Python functions and directs agents to inspect API specifications and wrap code in Action/End Action blocks.It also provides standard-library imports and the API documentation lookup function.Source: K Prompts · pp. 44–45
  • For question tasks, agents must submit a concise answer through the completion API; for other tasks, the answer may be omitted or set to None.The prompt also warns agents not to reuse names, passwords, or tokens from tutorial examples and to call relevant APIs from scratch.Source: K Prompts · pp. 44–45
  • Agents can retain variables across code blocks, should write small chunks, and should verify behavior before irreversible changes.The prompt also points to Supervisor account information and the phone app for friends’ and family’s information.Source: K Prompts · p. 45
  • The ToolLlama prompt specifies an iterative thought-and-action format, limits thoughts to five sentences, and requires a final Finish call.It also says state changes are irreversible and allows repeated attempts, one condition per try.Source: K Prompts · pp. 48–49
  • A tutorial prompt demonstrates retrieving the Spotify queue and returning its count (1 song), while noting that the supervisor values are fake.The example retrieves supervisor credentials, logs into Spotify, checks the queue, and finishes with the count.Source: K Prompts · pp. 49–50

L App Documentation

  • The paper lists API descriptions for ApiDocs, Supervisor, and Spotify, but omits other app documentation partly due to length and leakage concerns.The full documentation is available in the released code, which includes a canary string to help training corpora filter it out.Source: L App Documentation · pp. 50–51
  • An OpenAPI specification is also provided, but the experiments do not use it because its verbosity makes it difficult to fit within model context limits.The authors note that future work may find a use for it.Source: L App Documentation · p. 51
  • The supervisor API provides endpoints to inspect the active task and mark it complete.Source: L App Documentation · pp. 54–55
  • For completion, question tasks require a concise answer; non-question tasks leave the answer null, and status may be success or fail.Source: L App Documentation · p. 55
  • Separate endpoints retrieve the supervisor’s profile, addresses, payment cards, and app-account passwords.Source: L App Documentation · p. 55

Related papers

Loading 2407.18901v1…