PaperlayerView exact revision on arXiv

Source-linked AI summary

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, Niranjan Balasubramanian

arXiv:2407.18901v1cs.SEcs.AIcs.CLcs.LG

TL;DR

Existing benchmarks do not adequately test autonomous agents on everyday tasks requiring multi-app interaction, rich code, and reasoning over intermediate results. AppWorld introduces a controllable app-and-people simulator, a 750-task interactive coding benchmark, and state-based evaluation that accommodates valid solution variation and unintended changes. GPT4O with ReAct reaches 48.8 task goal completion on Test-N and 30.2 on Test-C, while the framework highlights open challenges for current LLMs.

  • Problem

    Existing tool-use benchmarks focus on short API-call sequences and reference-solution comparison rather than complex, interactive everyday tasks with many valid solutions.

  • Method

    AppWorld combines a controllable simulator of nine everyday apps and simulated people with 750 interactive coding tasks and programmatic database-state evaluation.

  • Results

    48.8 and 30.2 are GPT4O with ReAct task goal completion scores on Test-N and Test-C, while other evaluated models score lower.

  • Takeaways & Limitations

    AppWorld provides a rigorous environment and benchmark for assessing autonomous agents on complex API-based digital tasks and exposes substantial challenges for modern LLMs.

  • Takeaways & Limitations

    The benchmark has too few high-quality instances for training models and does not yet provide UI-based control for apps that expose functionality only through interfaces.

Abstract

from arXiv · show

Autonomous agents that address day-to-day digital tasks (e.g., ordering groceries for a household), must not only operate multiple apps (e.g., notes, messaging, shopping app) via APIs, but also generate rich code with complex control flow in an iterative manner based on their interaction with the environment. However, existing benchmarks for tool use are inadequate, as they only cover tasks that require a simple sequence of API calls. To remedy this gap, we built $\textbf{AppWorld Engine}$, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users. We then created $\textbf{AppWorld Benchmark}$ (40K lines of code), a suite of 750 natural, diverse, and challenging autonomous agent tasks requiring rich and interactive code generation. It supports robust programmatic evaluation with state-based unit tests, allowing for different ways of completing a task while also checking for unexpected changes, i.e., collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our 'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least 16% fewer. This highlights the benchmark's difficulty and AppWorld's potential to push the frontiers of interactive coding agents. The project website is available at https://appworld.dev/.

1 Introduction

AppWorld addresses the gap between simple tool-use benchmarks and everyday tasks that require multi-app interaction, rich code, and iterative reasoning. It combines a controllable app simulator with a challenging benchmark and robust evaluation, while current models achieve limited completion rates.

  • Motivation: Everyday digital tasks often span multiple applications and require interactive reasoning, sequential decisions, hurdle handling, and avoidance of undesirable outcomes.The paper illustrates this with grocery ordering and workout-playlist tasks that require inspecting intermediate results before writing subsequent code.
  • Motivation: Existing tool-use benchmarks mainly require linear sequences of 1–4 API calls and reference-solution matching, which is unsuitable for complex tasks with many valid solutions.These benchmarks generally do not require rich code or code generation based on environment interaction.
  • Framework: AppWorld combines a controllable simulator of real-world apps and people with complex tasks and evaluation suites built on that simulator.The Engine supports API operation without real-world consequences and models nine everyday apps with simulated activities from roughly 100 fictitious users.
  • Framework: 750 tasks require multi-app API workflows, rich iterative code, planning, failure handling, dynamic adaptation, and learning from interaction.Tasks use an average of 1.8 apps and 9.5 APIs, with average solution length of 50 lines and maxima of 6 apps, 26 APIs, and 134 lines.
  • Evaluation: The evaluation suite uses database-level unit tests to support multiple valid solutions while checking for unintended changes.Fine-grained database control enables robust, programmatic evaluation of goal completion and collateral damage.
  • Results: 48.8 and 30.2 are GPT4O with ReAct task goal completion scores on the normal and challenge test sets, respectively.GPT4Trb reaches 32.7 and 17.5, while GPT4O’s scenario-based robustness scores fall to 32.1 and 13.0.

2 AppWorld Engine

AppWorld Engine is a realistic, API-based simulator of everyday apps and people, designed to support safe, reproducible, and reliable agent interaction. Its database, APIs, execution shell, and procedural population process provide controllable digital activity at scale.

  • Applications: The Engine implements nine everyday apps with 457 documented APIs backed by database read/write operations, plus helper apps for API documentation and supervisor information.The apps span domains including email, money transfer, shopping, and local files; APIs average 50 per app and expose 1,470 arguments in total.
  • Execution Shell: Agents can write and execute stateful code snippets in a Jupyter-style shell using direct function calls or REST requests.The shell reuses variables, provides error traces and safe execution, and supports HTTP requests from any programming language.
  • Realism: The simulator models realistic app behavior, including consistent state changes, cross-app interactions, authentication, authorization, pagination, errors, and structured JSON outputs.For example, placing an order clears the cart and sends an email confirmation.
  • Control: Database and time control allow environments to be reset to identical starting conditions, customize task states, and construct robust programmatic evaluations.This control supports reproducibility across runs and time-dependent tasks.
  • Database: The Engine’s Base DB models digital lives and user activities across 101 tables, while rigorously tested APIs prevent inconsistent or unauthorized changes.The APIs provide higher-level operations such as placing an order and support time-consistent historical invocations.
  • Database: A procedural population method uses tested APIs and limited ChatGPT assistance to generate realistic, diverse activities for 106 fictitious people and roughly 370K database rows.Relationships and activities are designed to remain consistent with people’s roles and lives, while semantically important entries are manually generated.

3 AppWorld Benchmark

AppWorld Benchmark generates diverse, contrastive everyday tasks from scenario blueprints and evaluates them against database-state requirements. Its tasks include distractors, realistic hurdles, and multiple valid solution paths, while the challenge split tests generalization to unseen apps.

  • Task Generation: The benchmark targets natural, diverse, and challenging tasks covering many day-to-day scenarios.Task scenarios act as blueprints whose placeholder values and initial states are varied to instantiate multiple tasks.
  • Task Generation: Each task generator contains Setup, Evaluation, and Solution programs for instantiating tasks, defining assertions, and testing solvability end to end.Setup uses a scenario, Base DB, and base date and time to create task-specific instructions and starting states.
  • Task Generation: Setup programs enforce well-defined tasks, distractors, realistic hurdles, and contrast sets across tasks from the same scenario.Examples include multiple plausible past orders and an expired default payment card that requires an alternative route.
  • Evaluation: State-based evaluation accepts multiple valid final database states by checking expected changes, allowed non-mandatory changes, and the absence of other changes.This avoids reference-process matching and addresses both solution diversity and collateral damage.
  • Evaluation: Task Goal Completion measures the percentage of tasks passing all tests, while Scenario Goal Completion measures scenarios whose task variants all pass.SGC therefore assesses consistency across varying requirements and starting states.
  • Final Dataset: Test-C requires an API from at least one designated unseen app, whereas the remaining tasks are randomly split among Train, Dev, and Test-N.The unseen-app condition is intended to prevent memorized actions and require reading and following API documentation.
  • Final Dataset: 86% of tasks require a unique API set spanning 52 app combinations, and the benchmark is designed around planning, reasoning, interaction, hurdles, distractors, and rich code.The benchmark’s properties are reflected in its scenario construction and validation solutions.

4 Benchmarking Experiments

AppWorld experiments evaluate iterative prompting and code-based methods with closed and open LLMs, finding low completion across normal and challenge tasks. Performance declines with task difficulty, while oracle APIs provide only limited gains, indicating interactive code use and adaptation remain central challenges.

  • Main Results: All methods achieve low task and scenario completion scores across both Test-N and Test-C.ReAct with GPT4O is strongest, but completion remains limited.
  • Main Results: 48.8 TGC on Test-N falls to 30.2 on Test-C for ReAct + GPT4O.GPT4Trb reaches 32.7 and 17.5, while FullCodeRefl + LLaMA3 reaches 24.4 and 7.0.
  • Main Results: 30-50% lower scenario than task scores show that models do not consistently complete all task variants within a scenario.CodeAct and ToolLLaMA fail on all tasks, likely because of specialized narrow-domain training.
  • Difficulty Analysis: GPT4O performance drops as difficulty rises, with ReAct TGC decreasing from 58.3 at level 1 to 21.0 at level 3.Tasks requiring 60+ lines of code have TGC below 20, and similar trends occur for unique APIs.
  • API Retrieval: Oracle APIs improve TGC by up to 7.8 points on Test-N and 9.8 points on Test-C.The best Oracle API results are 54.8 and 35.2 TGC, respectively, alongside GPT4O API-predictor F1 scores of 87 and 71.
  • Error Analysis: The results suggest difficulty mainly reflects complex interactive API use, error adaptation, and other agent behaviors rather than API retrieval.Manual analysis identifies failures in interaction, API understanding, instruction following, commonsense, and state tracking.

5 Related Work

Prior benchmarks separately address tool use or executable-code generation, but AppWorld combines executable APIs with substantially more complex interactive coding tasks and programmatic evaluation.

  • AppWorld: AppWorld introduces interactive code generation for complex app-based tasks, combining tool usage with environment interaction.Its positioning addresses limitations at the intersection of prior tool-use and code-generation benchmarks.
  • Tool-Usage Benchmarks: Existing tool-use benchmarks typically provide no executable tools or simple public APIs and implementations, whereas AppWorld offers executable APIs for more complex tasks.AppWorld also provides more reliable and programmatic evaluation than existing approaches described here.
  • Tool-Usage Benchmarks: Existing evaluations often compare tool calls or answers with a gold solution using LLM or human judgment.Such comparisons do not fit tasks with multiple valid solution paths.
  • Interactive Code Generation Benchmarks: Code-generation benchmarks span short snippets, GitHub issue patches, interactive coding, and Python-interpreter reasoning tasks.These benchmarks differ from AppWorld in their task setting and tool environment.

6 Conclusions

The paper presents AppWorld as a controllable, reproducible environment and benchmark for interactive coding agents operating across complex everyday API tasks. Its evaluation and model analysis show that current systems still face major challenges automating such digital activities, while the engine supports broader extensions.

  • Conclusions: AppWorld addresses the lack of stable executable environments and sufficiently complex, interactive benchmarks for day-to-day autonomous agents.The framework combines a controllable execution environment, interactive coding tasks, and programmatic evaluation.
  • Conclusions: Programmatic evaluation, realistic hurdles, and distractors enable rigorous and robust assessment of autonomous agents.The evaluation supports the framework’s focus on complex API-based everyday tasks.
  • Future Directions: AppWorld Engine is modular and extensible toward UI control, multi-agent or human collaboration, and privacy and safety studies.These are presented as possible extensions and uses of the engine.

7 Limitations

AppWorld is limited to single-assistant API-based tasks, excludes direct UI interaction, and contains too few instances for model training. Its tasks also do not explore multi-agent coordination, while future extensions could add these capabilities.

  • Modalities: The benchmark covers API-accessible app functionality, but some real-world apps are available only through user interfaces.The authors suggest building a UI on top of the existing API backend to extend AppWorld to UI modalities.
  • Scope: AppWorld evaluates single-assistant tasks rather than multi-agent scenarios requiring coordination among assistants.The benchmark models one human supervisor assigning a task to one assistant; future work could extend the engine to multi-agent tasks.
  • Data scale: The carefully constructed benchmark contains too few instances for training models.The authors suggest bootstrapping or self-instruction for cheaper, noisier data augmentation.

8 Ethics Statement

The paper discusses risks from biased or potentially offensive generated content, limited cultural representativeness, and the hazards of deploying autonomous agents in the real world. AppWorld addresses some deployment risks through sandboxed experimentation and programmatic checks.

  • Bias: LLMs may reflect social, cultural, and ethical biases from their training data, even though AppWorld tasks avoid sensitive attributes.The authors also note that models built on state-of-the-art LLMs may exhibit these biases.
  • Representativeness: Benchmark tasks designed by North American and European authors may not reflect digital lives or needs in many other regions.The authors nevertheless regard enabling evaluation and development for these tasks as valuable from research and practical perspectives.
  • Deployment: Automating real-world tasks creates risks once deployed, while AppWorld provides a sandbox without real-world consequences for development and experimentation.The evaluation methodology also includes a systematic check, although the supplied passage truncates its details.
  • Data generation: A portion of the data was generated with ChatGPT under human review, but missed personally identifiable information or offensive content remains possible.The authors characterize this possibility as unlikely but not impossible.
  • Licensing: AppWorld’s APIs are original implementations, and the released system uses open-source, free-to-use tools under permissive licensing.The passage states that these implementations do not infringe the copyrights of corresponding real apps.
  • Documentation: Tasks with weak interaction requirements can become strongly interactive when response schemas are removed from API documentation.Agents must then inspect outputs to infer their structure, reflecting incomplete or outdated documentation in real APIs.

B AppWorld Benchmark Distributions

AppWorld Benchmark combines diverse, complex tasks with an execution shell for iterative API-based coding. Its distributions characterize task difficulty, solution complexity, programming constructs, and evaluation-test counts.

  • Programming constructs: Figure 6 counts tasks whose written solutions use constructs including datetime operations, data structures, numerical operations, exceptions, comprehensions, loops, conditionals, and regular expressions.The Y-axis is the number of tasks using each construct; nested loops are separately identified.
  • Core requirements: The shell is optional quality-of-life infrastructure because AppWorld fundamentally requires only HTTP calls, while database-state evaluation remains process-agnostic.The environment can therefore be accessed from any programming language capable of making those calls.
  • Execution shell: The execution shell supports stateful code execution, allowing agents to inspect outputs and reuse variables across code blocks.It is IPython-based and provides a notebook-like interaction pattern.
  • API access: Agents can call apps through either direct Python functions or REST requests, with the latter requiring HTTP methods, URLs, and parameter construction from documentation.Experiments use direct function calls for simplicity, although both interfaces are supported.
  • Runtime features: The shell offers best-effort safe execution, informative errors for failed requests and timeouts, frozen task-specific time, and a unified serverless process.Its safety features restrict systems-level modules, while SQLite and related components support the single-process setup.
  • Task distributions: Figure 5 reports task percentages by hand-labeled difficulty, validation-solution properties, and evaluation-suite test counts.The plotted solution properties include numbers of apps, unique APIs, and code lines.

D Hash-based Database Difference

AppWorld evaluates database changes by computing hierarchical start-to-end differences, using hashes and counters to avoid unnecessary comparisons. The benchmark also compares several interactive coding strategies under context-length constraints.

  • Database difference: Evaluation computes changes at table, row, and column levels between the task database’s start and end states.The resulting diff identifies changed tables, modified rows, additions or deletions, and changed columns.
  • Hash-based comparison: Per-row content hashes and per-table counters allow fast database differencing across 101 tables and 360K rows.A row hash changes when non-ID content changes, while table counters increment when record hashes change.
  • Optimization: The differencing procedure skips unchanged tables, then compares row hashes and IDs before inspecting column-level differences.This hierarchy saves work because unchanged tables do not require row- or column-level computation.
  • Runtime: Evaluation runs in under 0.6 seconds per task on average on a regular MacBook Pro.The speed is attributed in part to the hash- and counter-based comparison process.
  • Caveat: A table counter can miss a net change when modifications are made and then reverted.The authors explicitly note that a counter increment is not a guarantee that the table’s final state differs.
  • Modeling baselines: FullCodeRefl writes all code before execution, ReAct alternates reasoning and code, and PlanExec plans first before executing steps with ReAct-style trajectories.These three baselines are presented as distinct code-generation and execution patterns.
  • Documentation and history: ReAct and PlanExec explore API documentation through calls rather than receiving the full documentation in their prompts.PlanExec additionally supplies the executor with prior step history and code so variables can be reused.
  • Context management: Long interaction trajectories require truncating older environment observations while retaining recent outputs and task-specific context.The strategy targets verbose JSON observations and keeps the last two steps when a threshold is exceeded.

E.8 Cost of Experiments

The experiments were costly, limiting further exploration of modeling approaches and motivating future work on agent efficiency.

  • The experiments cost around $10K overall.This estimate includes different experiments, gold APIs, prompt variations, and the dataset size.
  • GPT4O cost about $0.7 per Test-N example for ReAct, compared with $1.33 for PlanExec, $0.33 for IPFunCall, and $0.02 for FullCodeRefl.These estimates use OpenAI rates current in June 2024.
  • GPT4Trb cost about twice as much as GPT4O, while GPT4 cost six times as much.
  • The expense limited further modeling exploration, and future work should improve agents’ cost, time, or token efficiency.

F.1 API Predictor performance

API retrieval is relatively strong, especially on Test-N, but it is not the primary source of AppWorld’s difficulty. Additional demonstrations help FullCodeRefl on Test-N but not Test-C, while performance consistently declines with task difficulty and model capability varies substantially.

  • API retrieval scores are relatively high for Test-N, although there remains room for improvement, especially on Test-C.
  • API retrieval is not the primary source of AppWorld Benchmark’s fundamental difficulty.
  • FullCodeRefl improves by up to 6.6 points on Test-N with more demonstrations but does not improve on Test-C.It still lags far behind ReAct.
  • GPT4O substantially outperforms GPT4 on Test-N, while Gemini Pro 1.5 underperforms LLaMA3 on Test-N.Gemini Pro 1.5 obtains TGC 20.2 and SGC 10.7, versus LLaMA3’s TGC 24.4 and SGC 17.9.

G Comparison with Web and UI Navigation Benchmarks

AppWorld differs from web and UI navigation benchmarks by evaluating interactive coding through APIs rather than executable screen actions. Figure 9 relates task distributions and GPT4O scores across hardness indicators for Test-C and Test-N.

  • Web and UI benchmarks evaluate executable actions such as clicking, typing, and scrolling across browser, mobile, or desktop interfaces.
  • Figure 9 shows Test-C in the top row and Test-N in the bottom row, with hardness indicators on the left axis and GPT4O scores on the right axis.The left column shows TGC scores and the right column shows SGC scores.
  • Methods perform worse as difficulty increases in both the TGC and SGC plots.
  • Interactive coding with APIs requires composing API invocations from documentation and execution outputs, unlike deciding which screen elements to manipulate.

H AppWorld Benchmark Examples

The benchmark examples illustrate how AppWorld’s scenarios and task instructions are organized, while the engine is supported by extensive API unit testing.

  • Tables 8 and 9 present additional AppWorld scenarios and their setup, while Table 10 gives task instructions from different hand-labeled difficulty levels.
  • AppWorld Engine’s APIs are validated with 1780 unit tests covering 98% of the API implementation code.Each API has approximately four unit tests on average, with examples from Spotify, Venmo, and File System apps.

J Task Generator Example

The task generator constructs a Spotify workout task by creating a varied, solvable initial state, embedding the workout duration in SimpleNote, and requiring a playlist to meet duration constraints. Its setup also introduces distractors and state changes that the validation solution must handle.

  • Task design: The generator designs tasks to be solvable, include distractors and hurdles, and form contrast sets for robust reasoning.These properties are explicitly described for scenario setup and task design.
  • Initial state: The setup creates a Spotify account state with multiple playlists, varied workout durations, and a renamed SimpleNote workout plan.The generated date is set to a weekday, the user must have enough playlists, and the note title is sampled from several alternatives.
  • Task construction: The task requires extracting the current day's workout duration from the note and selecting or constructing a playlist whose duration covers it.The setup maps weekday entries to durations, chooses a sufficiently long playlist, and builds other playlists within specified duration ranges.
  • State manipulation: The setup modifies Spotify state by removing and adding songs, clearing the player queue, and pausing the music player before evaluation.Songs are redistributed across playlists, while queued songs are removed and playback is paused without failing if already paused.
  • Validation solution: The gold solution logs into SimpleNote and Spotify, reads the workout plan, finds a playlist with sufficient total duration, plays it, and commits success.The task instruction explicitly requires uninterrupted playback during the workout, with the plan stored in SimpleNote.
Loading 2407.18901v1…