Environment-free Synthetic Data Generation for API-Calling Agents

Motivation

• Multi-step API-calling trajectories are difficult to collect because they must capture interleaved read and write calls, execution results, state tracking, and error recovery.Prior synthetic-data methods remain coupled to fully implemented APIs and realistic backend databases, creating an infrastructure and state-population bottleneck.Source: §1 · p. 1

Motivation and contribution

• ESAT eliminates the need to build executable environments or populate backend databases by generating trajectories using only API specifications.This reframes agentic trajectory generation as a lightweight specification problem that can scale across diverse API ecosystems.Source: §1 · pp. 1–2

Core mechanism

• A context-aware LLM API simulator uses task instructions and interaction history to generate coherent environmental feedback and state changes during multi-step API interactions.The simulator functions as a dynamic digital world model without real API execution.Source: §1 · pp. 1–2

ESAT pipeline

• The pipeline generates and filters tasks, has a teacher agent solve them through simulated API calls, then uses an LLM judge to retain correct, complete, and nonredundant trajectories.Task synthesis uses an LLM critic, while trajectory synthesis conditions simulation on the task and prior simulation history.Source: Figure 1 · p. 2, §1 · p. 2

Empirical validation

• Up to 50.5% on AppWorld and 60.5% on OfficeBench, ESAT fine-tuning improves models from 1.7B to 27B parameters over corresponding base models.On AppWorld, ESAT-trained models also outperform models trained on trajectories collected from the real AppWorld environment.Source: §1 · p. 2

Synthetic data with functional environments

• Functional-environment approaches ground trajectories through live endpoints or implemented sandboxes but require bespoke backends and diverse state data for each new API domain.This infrastructure dependence limits how quickly and broadly such methods scale.Source: §2 · p. 3

LLM as an API simulator

• LLM API simulators avoid executable backends, but implicit conversational coherence makes long trajectories vulnerable to state drift and hallucinated identifiers.ToolEmu targets safety evaluation, StableToolBench uses simulation for offline read-API evaluation, and Simia augments seed trajectories for SFT or simulated feedback for RL.Source: §2 · p. 3

ESAT novelty

• ESAT supports state-changing write APIs and maintains state consistency by validating each simulated response against the full interaction history.This persistent digital-world-model design enables multi-step interactions in dynamic, stateful environments.Source: §2 · p. 3
• Unlike prior pipelines requiring pre-compiled API pools, benchmark environments, or seed trajectories, ESAT generates tasks and trajectories from scratch using only API specifications.This makes the approach adaptable to new API domains without executable backends or pre-existing interaction data.Source: §2 · p. 3
• ESAT generates multi-step trajectories through explicit step-by-step interactions between separate agent and simulator LLMs rather than a single LLM pass.This interaction structure differs from the full-trajectory generation used by Li et al. (2025).Source: §2 · p. 3

3 Method

• ESAT transforms API specifications into datasets of multi-step API-calling trajectories through task synthesis, simulated execution, and LLM-based filtering.The pipeline requires neither executable APIs and backend infrastructure nor human-written tasks.Source: §3 · p. 3

3.1 Task Synthesis

• Task synthesis combines bucketized generation, inverse-frequency sampling, judge-based validation, and intent-level rewriting to produce diverse, balanced, solvable API tasks.Buckets vary difficulty, action type, task focus, app count, and API count; underrepresented apps and APIs are prioritized during generation.Source: §3.1 · pp. 3–4

3.2 Trajectory Synthesis

• A teacher agent solves each task through API calls while an LLM simulator generates stateful responses conditioned on specifications, arguments, task context, history, user details, and time.Simulator rules enforce consistency, diversity, realism, and schema compliance, while validation and judge-guided refinement reject or repair invalid responses.Source: §3.2 · p. 4

3.3 Trajectory Filtering

• Completed trajectories are retained only when repeated trajectory-level LLM judgments yield a majority-positive verdict that the task was correctly solved.Generation also stops on explicit task completion, execution-step limits, unsupported app usage, or simulation failure.Source: §3.3 · p. 5, §3.2 · p. 5

4.1 Results on AppWorld Benchmark

• AppWorld evaluation uses complex multi-app tasks requiring coordinated read/write API interactions, including held-out-app generalization in Test-Challenge.AppWorld contains 457 APIs across nine apps, with tasks averaging 10 unique APIs and 45 calls; Test-N has 168 tasks and Test-C has 417.Source: §4 · p. 5

4.1.1 Effectiveness of ESAT Data

• Synthetic apps generalize strongly: ESAT-S52 improves all AppWorld models by 8.4–47.0%, usually surpasses real-environment AWT, and combined ESAT-S52-AW7 adds up to 11.2%.ESAT-S52 uses 6K trajectories from 52 novel apps and 1017 APIs, while ESAT-AW7 uses 9K trajectories from 340 APIs across seven non-held-out apps.Source: §4.1 · p. 5, §4.1.1 · pp. 6–7, Table 1 · p. 6

4.2 Results on OfficeBench Dataset

• OfficeBench tests long-horizon office workflows, and ESAT-OB training uses 1.7K trajectories generated from its 20 APIs spanning eight apps.The evaluation reports pass rate over 26 two-app and 46 three-app tasks after excluding OCR-dependent and unsolvable tasks.Source: §4 · p. 5, §4.2 · p. 8
• Fine-tuning on ESAT data yields gains of 8.4–47.0% on AppWorld and 5.2–60.5% on OfficeBench, with smaller models sometimes outperforming much larger baselines.On AppWorld, several fine-tuned Qwen models surpass GPT-4o and Nemotron-3-120B; on OfficeBench, Qwen3-4B and Qwen3.5-4B also outperform those baselines, while ESAT exceeds Simia by at least 20%.Source: §4.1.1 · p. 6, Table 1 · p. 6, Table 2 · p. 7

4.3 LLM Judge Quality

• Gemini-3.1-Pro filters trajectories more effectively than GLM-5.1-FP8, achieving 95.2% precision against executable AppWorld correctness and 95% human agreement with Cohen’s kappa of 0.90.GLM filtering produced mixed downstream effects, whereas Gemini improved both evaluated metrics by more aggressively removing low-quality trajectories.Source: §4.3 · pp. 8–9

4.4 API Simulator Quality Analysis

• The simulator produced valid responses for 93.7% of approximately 27K calls, but failure rates increased with response length from 2–3% below 500 tokens to 23% beyond 2K.Input context length showed no significant correlation with failure rate; validation checked API specifications, arguments, and accumulated state.Source: §4.4 · pp. 9–10

4.5 Comparison with Existing Synthetic API-calling Datasets

• Compared with ToolAlpaca, ToolACE, and Nemotron, ESAT better supports complex stateful workflows: alternatives provide little downstream benefit, whereas ESAT yields substantial gains on both benchmarks.ESAT maintains consistent virtual state across calls, unlike turn-level or non-stateful simulators; even adding real-environment AWT leaves alternative-dataset models below ESAT-trained models.Source: Table 4 · p. 9, §4.5 · pp. 10–11, Table 5 · p. 10

• ESAT is an environment-free pipeline that synthesizes API-calling-agent training data using only API specifications.Source: §5 · p. 11
• ESAT combines coverage-aware task generation, a state-consistent LLM API simulator, and a trajectory-level LLM judge to produce high-quality multi-step trajectories without a functional environment.Source: §5 · p. 11
• On AppWorld and OfficeBench, ESAT-trained models achieve substantial performance gains across models ranging from 1.7B to 27B parameters.Source: §5 · p. 11
• On AppWorld, models trained on synthetic ESAT data outperform models trained on trajectories collected from the real environment.Source: §5 · p. 11
• On both benchmarks, 4B- to 8B-parameter models trained with ESAT either outperform or approach GPT-4o using significantly smaller models.Source: §5 · p. 11

Appendices

• The initial judge accepts 14,361 of 29,676 generated candidate tasks (48.39%), filtering out roughly 52% as ill-formed, unsolvable, inconsistent, or mismatched with requested bucket attributes.Table 6 reports task counts surviving each synthesis stage.Source: Table 6 · p. 14, §A.1 · p. 14
• The appendices report task and trajectory synthesis throughput alongside the distribution of trajectory-synthesis failure modes for ESAT-AW7.These summaries are presented in Tables 6–8.Source: Table 6 · p. 14, Table 7 · p. 14, Table 8 · p. 14

A.1 Synthesis Throughput and Yield

• 9,352 trajectories (65.13% of input tasks) survive quality filtering for supervised fine-tuning after 11,445 of 14,361 tasks are successfully completed.The teacher agent completes 79.70% of input tasks, with completion defined as terminating with a final answer regardless of correctness.Source: Table 7 · p. 14, §A.1 · p. 14
• 12,242 tasks (41.25% of the initial pool) pass the second judge after rewriting and re-validation against the original specifications.For rewritten tasks rejected by the second judge, the pipeline falls back to the corresponding initial tasks for trajectory synthesis.Source: §A.1 · p. 14
• The rewriting step reformulates 13,805 initially accepted tasks into concise, intent-level requests, discarding 556 tasks (1.87% of the original pool) because of rewriting failures.The rewritten tasks are subsequently checked by the second judge.Source: §A.1 · p. 14
• 15,037 of 245,762 simulated API calls (6.12%) require simulator retries, indicating that most responses pass schema and semantic validation on the first attempt.Retries address schema or semantic validation failures.Source: §A.1 · p. 14

Failure Analysis

• 45.38% of 5,009 discarded trajectories result from simulation failures, mainly simulator timeouts or responses failing schema and consistency checks within the retry budget.Timeouts occur when agents issue too many simulated API calls in one step; other failures arise when valid responses cannot be generated within the retry budget.Source: §A.2 · p. 14
• 41.78% are judge rejections: trajectories complete end-to-end but are filtered for incorrect or low-quality solutions as intended quality control.Together with simulation failures, judge filtering and simulator-side validation account for roughly 87% of all discarded trajectories.Source: §A.2 · pp. 14–15
• The remaining failures are minor: unexpected App use accounts for 10.72%, model execution failures for 2.00%, and both max solve steps exceeded and runtime errors for 0.06%.Unexpected App use triggers early termination, execution failures reflect transient serving-stack errors, and max-step failures occur when agents exceed the step budget.Source: §A.2 · pp. 14–15

A.3 Dataset Diversity and Coverage

• Generated tasks provide fairly uniform coverage across difficulty, action type, and task focus, but two- and three-app tasks are significantly less frequent than single-app tasks.The skew may reflect the difficulty of composing meaningful multi-app tasks and naturally non-uniform real-world task distributions.Source: §A.3 · p. 15
• Successfully generated trajectories show broad coverage across difficulty levels, action types, and task focus categories, with fewer two- and three-app tasks than single-app tasks.Thus, trajectory diversity follows the task distribution while retaining the same multi-app imbalance.Source: §A.3 · p. 15
• 339 of 340 AppWorld APIs (99.7%) appear in at least one successful trajectory, while 323 (95%) are used at least five times.The coverage analysis spans frequency thresholds from 1 to 50 successful trajectories, as shown in Fig. 4.Source: §A.3 · p. 15, Figure 4 · p. 16

B.1 Synthesis Throughput and Yield

• 7,072 curated tasks remain from 32,009 generated tasks, a final retention rate of 22.09% after judging and rewriting.The first judge retains 10,509 tasks (32.83%), rewriting produces 9,171 successful reformulations, and a secondary judge performs the final cross-check.Source: §B.1 · p. 16
• 6,265 high-quality trajectories remain from 10,509 input tasks, retaining 59.62% for supervised fine-tuning after quality filtering.The teacher completes 8,020 trajectories (76.32%) before the judge removes lower-quality outputs.Source: §B.1 · p. 16
• Only 6,062 of 154,317 simulated API calls require retries, a 3.93% retry rate caused by schema or semantic validation errors.The low retry rate indicates that the simulator usually produces well-formed and consistent responses on the first attempt.Source: §B.1 · p. 16

Failure type of trajectory generation

• 28.30% were simulation failures, mainly caused by simulator timeouts or inability to satisfy schema and consistency checks within the retry budget.Timeouts occurred when agents issued too many simulated API calls in one step.Source: §B.2 · p. 16
• 22.05% involved unexpected App use, 8.25% involved teacher-model execution failures, and max-step or runtime failures each accounted for 0.02%.Unexpected App use triggered early termination, while execution failures arose from serving-stack issues such as rate limits or decoding errors.Source: §B.2 · pp. 16–17
• 41.35% of 4,244 discarded trajectories were judge rejections, representing quality-control filtering of incorrect or low-quality end-to-end solutions.These trajectories completed end-to-end but were filtered by the LLM judge.Source: §B.2 · p. 16, Table 11 · p. 17

B.3 Dataset Diversity and Coverage

• Generated tasks broadly cover task focus, action types, and difficulty, but single-app tasks dominate at 64%, versus 26% two-app and 11% three-app tasks.The multi-app skew may reflect the difficulty of composing meaningful multi-app tasks and naturally non-uniform real-world task distributions.Source: §B.3 · p. 17, Figure 5 · p. 18
• Successful trajectories span difficulty levels—easy 31%, medium 31%, hard 38%—and action types—mixed 38%, read 32%, write 29%.Task-focus coverage is also broad: constraint satisfaction 25%, derivation 21%, iteration 28%, and open 27%.Source: §B.3 · p. 17
• Single-app trajectories comprise 83% of successful trajectories, while two-app and three-app trajectories account for only 14% and 3%, respectively.Thus, the trajectory distribution retains broad capability coverage but remains strongly skewed toward single-app interactions.Source: §B.3 · p. 17
• 1,013 of 1,017 APIs (99.6%) appear in at least one successful trajectory, and 951 (93.5%) appear at least five times, indicating near-complete API coverage.Inverse-frequency sampling prevents concentration on a narrow set of popular endpoints.Source: §B.3 · p. 17, Figure 6 · p. 18

C.1 Synthesis Throughput and Yield

• 2,380 of 11,797 candidate OfficeBench tasks survive final validation, yielding 20.17% of the initial pool after judges and rewriting filter generations.The initial judge accepts 3,465 tasks (29.37%), rewriting succeeds for 3,336, and rejected rewritten tasks fall back to their initial tasks.Source: §C.1 · pp. 18–19, Table 12 · p. 19
• 1,711 trajectories survive quality filtering from 3,465 input tasks, a 49.38% yield, while only 919 of 12,084 simulated API calls require retries (7.61%).The teacher completes 1,889 trajectories (54.52%), and the low retry rate indicates mostly first-attempt simulator reliability.Source: Table 13 · p. 19, §C.1 · p. 19

C.2 Failure Analysis

• 77.42% of discarded trajectories result from agent model execution failure, followed by judge rejection at 10.15%, simulation failure at 8.61%, and step-budget exhaustion at 3.82%.The 30-step limit captures loops or tasks requiring longer action sequences, while infrastructure-related failures form smaller shares.Source: Table 14 · p. 19, §C.2 · p. 19

Task distribution

• Generated tasks cover task focus, action type, and difficulty nearly uniformly, while two-app tasks form the largest app-count category at 40%.Single-app and three-app tasks comprise 35% and 25%, respectively, reflecting an emphasis on complex multi-app orchestration.Source: §C.3 · pp. 19–20

Trajectory distribution

• Trajectory distributions are balanced across difficulty, action type, and task focus, with easy, medium, and hard tasks at 34%, 32%, and 34%, respectively.Mixed read/write actions lead at 43%, while read and write actions account for 30% and 27%; iteration and open tasks are most common at 29% and 28%.Source: §C.3 · p. 20

API coverage

• 70% of the 20 available APIs appear in successful trajectories, with 70% used at least five times and 60% used at least 50 times.Coverage declines only to 65% at the 10-use threshold; six APIs are absent, while the remaining APIs cover most of the OfficeBench action space.Source: §C.3 · p. 20

Data synthesis

• Data synthesis uses GLM-4.7-FP8 for task generation and validation, GLM-5.1-FP8 for teaching and API simulation, and Gemini-3.1-Pro for trajectory filtering.Trajectory lengths are capped at 50 agent steps for AppWorld and 30 for OfficeBench; over-limit trajectories are aborted.Source: D Implementation Details · p. 20

Training

• Models are fine-tuned on full multi-turn trajectories with masked context loss, AdamW at 2 · 10−5, 32,768-token contexts, sequence packing, and distributed GPU training.Training lasts 10 epochs by default, except Qwen3.5 models, which use 5 epochs because of overfitting; AppWorld uses split-specific checkpoint selection, while OfficeBench reuses AppWorld hyperparameters without a development set.Source: D Implementation Details · p. 21

Inference and evaluation

• Evaluation uses vLLM with temperature 1.0, benchmark-specific step caps, eight independent samples per task, and average pass@1 reporting.The step limits are 50 for AppWorld and 30 for OfficeBench.Source: D Implementation Details · p. 21

E Training with AppWorld Train splits

• The comparison uses 90 AppWorld training tasks, eight teacher trajectories per task, filtering incorrect solutions before optionally duplicating valid trajectories to eight.Four Qwen variants are fine-tuned under both settings and evaluated on Test-N and Test-C.Source: E Training with AppWorld Train splits · p. 21
• Upsampling improves Qwen3.5-9B Test-N accuracy (73.14 vs. 71.21) but lowers Test-C accuracy (51.89 vs. 53.84), while the other models decline on both splits.The largest gap is on Qwen3.5-4B Test-C (42.39 vs. 46.70).Source: Table 15 · p. 22, E Training with AppWorld Train splits · p. 22
• Because naive duplication can reduce sample diversity and amplify tasks with few valid trajectories, the authors adopt non-upsampling thereafter.The reported results indicate that balancing per-task trajectory counts can hurt generalization.Source: E Training with AppWorld Train splits · p. 22

AppWorld

• AppWorld results cover Task Goal Completion and Scenario Goal Completion across the Test-Normal and Test-Challenge splits.Table 16 reports means and standard deviations across eight runs; TGC denotes Task Goal Completion and SGC denotes Scenario Goal Completion.Source: F Full Experimental Results · p. 22

OfficeBench

• OfficeBench results break down full pass rates for 2-app, 3-app, and overall tasks.Table 17 reports mean performance with standard deviations across eight independent inference runs.Source: F Full Experimental Results · p. 22, Table 17 · p. 24

Evaluation protocol

• On the AppWorld and OfficeBench benchmarks, results for all evaluated models and training-data configurations are reported as mean performance with standard deviations across 8 independent inference runs.Source: F Full Experimental Results · p. 22

G Task Synthesis

• Task synthesis uses API specifications to generate, judge, and rewrite diverse natural multi-step tasks, with GLM-4.7-FP8 (Zeng et al., 2025) serving all three roles.The specifications provide API names, purposes, input schemas, and output schemas.Source: G Task Synthesis · p. 24

G.1 Bucketized generation.

• The generator spans 360 configuration buckets combining difficulty, action type, task focus, app count, and per-app API budgets.Buckets are processed in random order, each targeting a fixed number of accepted tasks.Source: §G.1 · p. 25

G.2 Inverse frequency-based sampling.

• Inverse-frequency sampling prioritizes underused apps and APIs while preserving access to each selected app’s full specification.Apps are sampled with p(a) ∝1/(1 + c(a)), and each selected app exposes its top-10 inverse-coverage APIs as focus APIs.Source: §G.2 · p. 25

G.3 Task judging.

• An LLM judge simulates solving each candidate task and retains it only when API coverage, difficulty, focus, and other stated requirements are satisfied.The judge receives the same API specifications and requirements block shown to the generator and returns a yes-or-no verdict.Source: §G.3 · p. 25

G.4 Task rewriting.

• A rewriter converts procedurally verbose tasks into concise intent-level requests while preserving required details and API coverage.It sees the original task, shared requirements, covered API specifications, and three verbose-to-compact in-context examples; Table 18 illustrates the compression.Source: §G.4 · pp. 25–26, Table 18 · p. 26

G.5 Final re-judging.

• Every rewritten task undergoes a second judge pass, and only tasks passing both rounds enter the final pool.Rejected rewrites retain their original generation metadata for diagnostics but are excluded from the pool.Source: §G.5 · p. 26

G.6 Prompts for Task Synthesis

• The synthesis prompts instantiate sampled difficulty, action-type, and task-focus clauses, while separate templates specify generation, judging, and rewriting instructions.Tables 19–21 enumerate the clause variants, and Listings 1–4 provide the corresponding prompt templates and examples.Source: §G.6 · pp. 26–27, Table 19 · p. 27, Table 20 · p. 27

H Simulator Details

• The simulator is text-only, with no backend, database, or network call, and produces JSON responses that follow schemas, remain task-consistent, and reflect the user profile.Responses are conditioned on the teacher agent’s API call and all prior responses in the same task.Source: H Simulator Details · p. 27

H.1 Per-Task, Per-App Histories

• Per-task, per-app histories isolate simulated state, cap prompt length, include recent calls and responses, and clear completely when tasks finish.This prevents irrelevant entities and state from leaking across applications or tasks.Source: §H.1 · p. 28

H.2 Deterministic Input Validation

• Deterministic input validation rejects unknown or missing parameters, type mismatches, and constraint violations before invoking the LLM, producing repeatable backend-like errors.Checks cover scalar coercions, container types, bounds, enumerations, length limits, and email formats.Source: §H.2 · p. 28

H.3 Prompt Construction; H.7 Prompt for Simulator

• Each simulator call receives the API, normalized inputs, full and response-specific schemas, task, per-app history, user profile, and frozen date-time, and must return raw JSON.The simulator and judge share inlined simulation rules, while prompt templates provide the corresponding call and evaluation context.Source: §H.3 · p. 29, §H.7 · pp. 29–30

H.4 Output Schema Validation

• Responses are stripped of code fences, strictly parsed, and accepted only when matching either the documented success schema or canonical error schema; otherwise structured feedback triggers retries.This output validation applies to every LLM response before semantic evaluation.Source: §H.4 · p. 29

H.5 Simulator-judge

• Semantic judging is applied only to read-only APIs, while structurally valid writes are accepted directly to avoid disproportionate extra LLM cost.The read-API judge receives the same context as the simulator and returns a validity flag, issues, and suggestions.Source: §H.5 · p. 29

H.6 Retry Loop With Structured Feedback

• A dual-budget retry loop feeds rejected responses and precise format or semantic feedback back into the conversation, terminating synthesis when either budget is exhausted.The authors report convergence within budget for the vast majority of calls.Source: §H.6 · p. 29

I Agent; J Trajectory Filtering

• The system defines separate AppWorld and OfficeBench agent prompts and trajectory judges, adapting completion signals, action formats, app switching, and side-effect grading to each benchmark.AppWorld filtering checks nine requirements, whereas OfficeBench evaluates JSON action blobs and file-based task effects.Source: I Agent · p. 30, J Trajectory Filtering · p. 30

K Synthetic Application Suite; K.1 Domain Selection; K.2 API Schema Design; K.3 Schema Conversion; K.4 AppWorld Integration

• The synthetic suite contains 52 REST-API apps across ten manually curated domains, with 1,017 endpoints generated from structured examples, verified against 13 invariants, converted to AppWorld format, and deployed as separate modules.Canary strings provide source traceability and memorization-detection signals; deterministic conversion fallback ensures valid output for all 52 applications.Source: K Synthetic Application Suite · p. 31, §K.1 · p. 31, §K.2 · pp. 31–32

K.4 AppWorld Integration

• Task prompts generate multi-step API-solving tasks and rewrite them to preserve intent, required APIs, explicit constraints, and crucial identifiers while removing redundant procedural wording.The rewriting process also reports which APIs the rewritten task covers and uses examples to retain conditions such as file paths, actions, names, dates, and amounts.Source: §K.4 · pp. 34–36
• The simulator generates raw JSON responses that match exact schemas and remain realistic and consistent with input parameters, task context, user profile, current time, and previous API history.Rules cover user-specific data, invalid-call errors, pagination, modifications, aggregate fields, deletion, timestamps, numeric constraints, unique identifiers, and plausible values.Source: §K.4 · pp. 37–38
• List simulation follows page limits and history-dependent pagination, while returning fewer items when realistic and preserving remaining items across subsequent pages.A later page is generally empty unless prior history indicates additional items, including items added by subsequent API calls.Source: §K.4 · pp. 37, 40
• Simulation data deliberately mixes task-relevant and plausible irrelevant entities, targeting roughly 30-50% task-relevant items in list results to model realistic retrieval.The rules also require unique identifiers, chronologically valid timestamps, constrained numeric values, and realistic names and contact details.Source: §K.4 · p. 38
• A separate judge evaluates each simulated API response against the API documentation, schema, call inputs, task, history, user information, time, and simulation rules, returning validity, issues, and repair suggestions as JSON.The judge accounts for page_limit behavior and requires empty issues and suggestions when a response is valid.Source: §K.4 · pp. 40–41
• AppWorld prompts expose app and API documentation, preserve variables across code-execution steps, require authentication and documentation lookup, and enforce pagination and task-completion rules.Agents must use the prescribed API namespace, avoid unauthorized changes, read referenced sources, determine dates programmatically, and complete tasks with or without an answer as appropriate.Source: §K.4 · pp. 42–43
• The AppWorld trajectory judge checks documentation adherence, contextual reasoning, logical strategy, chained information flow, grounded parameters, complete iteration, correct completion, and avoidance of duplication or irrelevant actions.The judge evaluates synthetic outputs from the trajectory rather than external factual knowledge, while allowing minor inefficiencies.Source: §K.4 · p. 46
• OfficeBench uses system actions to switch among apps and finish tasks, while restricting file access to each task’s testbed data directory.Question tasks return literal answers; non-question tasks return "None", and trajectories use structured JSON actions with observations.Source: §K.4 · p. 48
• The OfficeBench judge requires demonstrated completion, correct grounded answers, required file or communication side effects, and source-based values rather than invented information.It verifies that requested files, emails, events, and extracted content are produced from the task, observations, or permitted apps.Source: §K.4 · p. 48
• The judge prioritizes the correct end-state and visible trajectory evidence, while allowing minor inefficiencies such as extra reads or unnecessary app switches.Wrong-app errors may be recovered through switch_app, but persistent thrashing, skipped required reads, fabricated outputs, invalid JSON actions, and unsafe file manipulation are disallowed.Source: §K.4 · p. 48
• The OfficeBench trajectory judge evaluates whether an agent completely solves the task using eight requirements tailored to JSON actions and file-side-effect grading.The judge receives the task and interaction trace, explains its decision, and outputs YES or NO; OfficeBench exposes only first_name in the task header.Source: §K.4 · pp. 48–49
• Gemini is prompted with curated domain entries and representative APIs to invent distinct apps supporting at least 30–50 APIs.The prompt requires CRUD resources, nested sub-resources, stateful actions, search and filtering, social features, notifications, payments, and file downloads.Source: §K.4 · pp. 49–51
• When generating multiple apps, the prompt enforces completely different domains and forbids reusing well-known brand names.This diversity constraint is appended after previously generated app names are supplied to Gemini.Source: §K.4 · p. 51
• The generated app specification is a JSON object containing app metadata and APIs with paths, methods, descriptions, parameters, and response schemas.Conventions include access tokens for non-public APIs, ten required account-management APIs, pagination for list APIs, and structural few-shot examples.Source: §K.4 · p. 51

Task decomposition

• The agent must find cards expiring before 2027, select the latest-expiring one, conditionally transfer funds based on recent activity and balance, and then recheck the balance.The task specifies a 90-day transfer-count threshold of fewer than 3 and conditional transfers of $50 and an additional $100.Source: L Example Agent Trajectories · p. 52

API discovery

• It begins by inspecting the banking app’s available APIs before planning the required operations.The API list includes balance, transfer-history, and payment-card operations relevant to the task.Source: L Example Agent Trajectories · pp. 52–54

Authentication

• The agent retrieves banking-app credentials, logs in with the required username and password, and obtains a bearer access token.The login specification requires account email and password and returns an access token; the supervisor app supplies the banking-app password.Source: L Example Agent Trajectories · pp. 55–56

Card selection

• Using the token, it lists four payment cards and selects Citi Double Cash, card ID 3, as the latest-expiring card among those expiring before 2027.The qualifying cards expire in 2024, 2025, and 2026, while the fourth card expires in 2028.Source: L Example Agent Trajectories · p. 57

Transfer-history check

• The next step is to query bank-transfer history for the selected card, filtering transfers by direction and potentially by pagination.The transfer-history API supports banking-app-to-card filtering and page index and limit parameters, but this excerpt does not provide the returned history.Source: L Example Agent Trajectories · pp. 57–58

Example trajectory 1

• The trajectory finds 2 transfers to the Citi Double Cash card in the last 90 days, recognizes that 2 is less than 3, and initiates a $50 withdrawal.The withdrawal succeeds, the reported banking-app balance is $125, and a subsequent withdrawal also succeeds.Source: L Example Agent Trajectories · pp. 60–61
• The agent iteratively explores API documentation and chains multi-step calls, with all API responses generated by the LLM-based simulator.The trajectory culminates in supervisor.complete_task() after the banking operations succeed.Source: L Example Agent Trajectories · pp. 60–61, Figure 9 · p. 61

Example trajectory 2

• A second trajectory asks the agent to save the reviewer’s name and rating from the most recent review of its highest-rated liked album to a file.The agent begins by discovering available applications and then inspects music and file-storage APIs, including liked-album and album-related functionality.Source: L Example Agent Trajectories · pp. 62, 64

• The agent obtains the music app password through the supervisor API, then uses the returned access token for subsequent calls.It calls show_account_passwords and identifies the music_app account before receiving a bearer access token.Source: L Example Agent Trajectories · pp. 67–68
• The agent consults the show_liked_albums specification to determine its endpoint, required token, pagination parameters, sorting options, and response schema.The API supports page_index, page_limit, and sorting by liked_at or title.Source: L Example Agent Trajectories · pp. 68–69
• It iterates through result pages, accumulates liked albums, stops when a page contains fewer than ten results, and reports the total count.The trajectory explicitly initializes pagination, extends the album list, and breaks when len(response) < 10.Source: L Example Agent Trajectories · p. 69
• The agent selects the liked album with the highest rating, identifying “Midnight Dreams” with a rating of 4.9.The selection uses the rating field as the maximization key.Source: L Example Agent Trajectories · pp. 69–70
• It then consults the show_album_reviews API specification to retrieve reviews for the selected album using its album ID and optional filters.The specification includes album_id, query, user_email, minimum and maximum rating, and pagination parameters.Source: L Example Agent Trajectories · p. 70

Example trajectory 2

• The demonstrated trajectory retrieves the newest review for “Midnight Dreams” by sorting album reviews by creation date in descending order and limiting the page to one result.The returned review has album_review_id 5678, album_id 101, and rating 5.0.Source: L Example Agent Trajectories · p. 71
• The file-writing step completes successfully, after which the agent marks the task complete through the supervisor API.The response states “File created successfully” and returns the created file path.Source: L Example Agent Trajectories · p. 74
• The agent then inspects file-storage documentation, authenticates, and creates ~/reviewer_rating.txt containing the reviewer’s name and rating.The API requires a file path and access token; the simulated response confirms successful creation at /home/user/reviewer_rating.txt.Source: L Example Agent Trajectories · pp. 72–73, Figure 10 · p. 74

Example trajectory 3

• A second example begins by exploring the action-item-list API documentation to identify operations for creating projects, labels, and tasks.The listed APIs include project, task, sub-task, and label creation and retrieval operations.Source: L Example Agent Trajectories · pp. 75–76

Project and label setup

• It creates a ‘Debt Tracking’ project and a ‘Settle Up’ label, receiving project_id 482910 and label_id 892347.The simulated responses report successful creation of both resources.Source: L Example Agent Trajectories · pp. 78, 79, 82

Task and label chaining

• After finding that create_task does not support labels directly, the agent consults add_label_to_task and plans to create tasks before applying the label separately.The task API supports project_id and title plus optional scheduling and priority fields, while add_label_to_task requires task_id and label_id.Source: L Example Agent Trajectories · pp. 80–81

Example trajectory

• The agent iteratively explores API documentation, authenticates, and chains multi-step API calls, with all responses generated by the LLM-based simulator.Figure 11 presents this trajectory as an example of solving the user task through simulated API interactions.Source: Figure 11 · p. 82