PaperlayerView exact revision on arXiv

Source-linked AI summary

Environment-free Synthetic Data Generation for API-Calling Agents

Seanie Lee, Sanjoy Chowdhury, Chao Jiang, Cheng-Yu Hsieh, Ting-Yao Hu, Alexander T Toshev, Oncel Tuzel, Raviteja Vemulapalli

arXiv:2607.16900v2cs.AI

TL;DR

Training API-calling agents requires costly multi-step trajectories and executable environments, limiting scalable data collection. ESAT generates trajectories from API specifications using LLM-synthesized tasks, simulated feedback, and judging, and fine-tuning on its data improves performance on AppWorld and OfficeBench.

  • Problem

    High-quality multi-step API-calling trajectories are difficult to collect at scale, while existing synthetic-data methods require executable APIs and populated backend environments.

  • Method

    ESAT uses only API specifications to synthesize tasks, simulate stateful API feedback during teacher-agent interactions, and filter trajectories with LLM critics and judges.

  • Results

    Fine-tuning on ESAT data substantially improves performance across AppWorld and OfficeBench, reaching up to 50.5% on AppWorld and 60.5% on OfficeBench.

  • Takeaways & Limitations

    ESAT shows that effective API-calling-agent supervision can be generated without functional environments across diverse API ecosystems.

  • Takeaways & Limitations

    Simulation failure increases with response length, reaching 23% beyond 2K tokens.

Abstract

from arXiv · show

Training API-calling large language model (LLM) agents demands massive amounts of high-quality trajectories. However, collecting such data at scale typically requires fully implemented environments with executable APIs and realistic, pre-populated backend databases, creating a major bottleneck for scalability. To overcome this, we propose an environment-free synthetic data generation approach that leverages LLMs as on-the-fly digital world models. Given only API specifications, our method generates trajectories mimicking interactions between an agent and a stateful environment. Specifically, an LLM first generates diverse tasks solvable with the provided APIs. A teacher agent then iteratively solves each task while an LLM simulator generates coherent synthetic API responses conditioned on the task context and simulation history. Finally, an LLM judge filters the trajectories to ensure the quality of the resulting dataset. We evaluate our approach on the challenging AppWorld and OfficeBench benchmarks, which include both information-retrieval and state-changing tasks. Fine-tuning models on our synthetic data yields significant performance gains, demonstrating that effective supervision for API-calling agents can be generated without any executable environment. Our results establish LLM-based API simulation as a practical, scalable solution for training agents across diverse API ecosystems.

1 Introduction

ESAT addresses the bottleneck of training API-calling agents by generating coherent, stateful trajectories from API specifications without executable environments. Its pipeline synthesizes and filters tasks and trajectories, and improves fine-tuned agents on AppWorld and OfficeBench.

  • Motivation: Training API-calling agents requires extensive high-quality trajectories covering multi-step interactions with interleaved API calls and environmental feedback.Existing synthetic-data methods remain coupled to fully implemented APIs and realistic backend databases.
  • Core approach: ESAT uses a context-aware LLM API simulator to generate coherent environmental feedback and state changes without executing real APIs.The simulator conditions responses on API specifications, task context, input arguments, and prior interaction history.
  • Pipeline: The pipeline generates diverse API-solvable tasks, filters invalid tasks, has a teacher agent solve them through the simulator, and judges trajectories for quality.Figure 1 describes task generation and filtering, teacher-agent interaction with the API simulator, and final trajectory selection.
  • Advantages: ESAT reduces trajectory synthesis to an API-specification problem, eliminating infrastructure and database bottlenecks while supporting scale across diverse API ecosystems.Its long-context simulator maintains state coherence across successive calls, enabling rich multi-turn trajectories.
  • Evaluation: Up to 50.5% on AppWorld and 60.5% on OfficeBench, supervised fine-tuning on ESAT data substantially improves models from 1.7B to 27B parameters over corresponding base models.The evaluation covers information-retrieval and state-changing tasks on both multi-app benchmarks.

2 Related Work

Related work synthesizes agent trajectories either by executing tool calls in functional environments or by using LLMs to simulate API responses. ESAT instead supports state-changing APIs and generates tasks and trajectories from scratch using only API specifications.

  • Synthetic data with functional environments: Recent synthetic-data methods execute generated tool calls against live API endpoints or fully implemented environments to improve trajectory fidelity.These approaches use LLMs to synthesize agentic trajectories at scale because human-annotated interaction data is resource-intensive.
  • LLM as an API simulator: ToolAlpaca, ToolACE, and Nemotron prompt LLMs to generate mock API execution results for multi-turn synthetic data.The passage notes that relying on implicit conversational coherence leaves such methods vulnerable to state drift and hallucinated information.
  • ESAT’s differences from prior simulation pipelines: ESAT supports state-changing write APIs, whereas most prior simulators focus on read APIs, enabling interactions in dynamic, stateful environments.It also generates both tasks and trajectories from scratch using only API specifications, unlike methods requiring pre-compiled API pools, benchmarks, or seed trajectories.

3 Method

ESAT converts API specifications into multi-step API-calling trajectories without real API execution, backend infrastructure, or human-written tasks. Its pipeline synthesizes balanced tasks, simulates stateful API interactions with a teacher agent, and filters results using LLM judges.

  • Pipeline overview: ESAT comprises task synthesis, teacher-agent trajectory synthesis with an LLM API simulator, and LLM-based trajectory filtering.The pipeline requires no real API execution, backend infrastructure, or human-written tasks.
  • Task synthesis: Task generation uses configuration buckets spanning difficulty, action type, task focus, app count, and APIs per app to produce diverse, balanced coverage.Buckets are processed in randomized order, with sampled app specifications and a target number of tasks per bucket provided to the generator LLM.
  • Task synthesis: Inverse-frequency sampling prioritizes underrepresented apps and APIs during task generation.Running usage counts are maintained across generated tasks, and app-selection probabilities are set inversely proportional to those counts.
  • Task synthesis: Accepted tasks are LLM-judged for validity, rewritten from procedural instructions into concise intent-level requests, and verified again.The initial judge checks that tasks are well-formed, solvable, and consistent with their bucket configuration.
  • Trajectory synthesis: A teacher agent issues API calls while the simulator generates responses conditioned on API specifications, arguments, task context, interaction history, virtual-user details, and current time.This produces complete multi-step trajectories containing reasoning, API calls, and simulated environmental observations without executing real APIs.
  • Trajectory filtering: Completed trajectories are retained only after repeated trajectory-level LLM judgments yield a majority-positive verdict.Only trajectories ending with the explicit task completion API are evaluated by this final judge.

4 Experiments

Experiments on AppWorld and OfficeBench show that environment-free ESAT synthetic data substantially improves API-calling performance, generalizes to unseen APIs and user states, and can outperform larger baselines. Additional analyses validate the simulator and judge while identifying longer-response simulation as a limitation.

  • Benchmarks: AppWorld tests complex read/write coordination across 457 APIs and nine apps, while OfficeBench evaluates long-horizon workflows over 20 APIs and eight apps.OfficeBench evaluation included 26 two-app and 46 three-app tasks after excluding OCR-dependent and unsolvable tasks.
  • Training data: ESAT-S52 contains 6K trajectories generated from 52 synthetic apps with 1017 APIs spanning diverse domains independently of AppWorld.The synthetic apps covered e-commerce, productivity, travel, entertainment, and finance.
  • Generalization: 8.4–47.0% gains across all models show that fine-tuning on ESAT-S52 transfers API-calling skills to unseen APIs.The ESAT-S52 apps were independent of AppWorld, providing a direct test of API generalization.
  • Knowledge transfer: Qwen3 8B and 14B plus Qwen3.5 4B, 9B, and 27B outperform GPT-4o and Nemotron-3-120B after ESAT-S52-AW7 fine-tuning, while Qwen3.5-27B comes within < 3% of GLM-5.1-FP8.These results demonstrate knowledge transfer from the larger teacher to substantially smaller models.
  • Downstream performance: 5.2–60.5% gains on Table 2 confirm that environment-free synthetic data provides effective supervision, with smaller Qwen models outperforming Nemotron-3-120B and GPT-4o.The experiments also report that ESAT outperforms the recent Simia dataset.
  • Validation and limitations: 95% overall agreement and Cohen’s kappa of 0.90 show that the LLM judge closely matches executable AppWorld correctness, while simulator validity reaches 93.7% of calls.The judge disagreed on only 8 accepted and 2 rejected trajectories; simulator failures increase from 2–3% below 500 tokens to 23% beyond 2K tokens.

5 Conclusion

ESAT is an environment-free pipeline that uses only API specifications to synthesize high-quality, multi-step training trajectories for API-calling agents. Its coverage-aware task generation, state-consistent LLM API simulation, and trajectory-level judging produce substantial gains across AppWorld and OfficeBench models from 1.7B to 27B parameters.

  • 5 Conclusion: ESAT synthesizes training data from only API specifications, without using any functional environment.The pipeline targets API-calling agents and produces multi-step trajectories.
  • 5 Conclusion: Coverage-aware task generation, state-consistent LLM API simulation, and trajectory-level LLM judging enable ESAT to produce high-quality, multi-step trajectories.These three components are combined within the ESAT pipeline.
  • 5 Conclusion: ESAT evaluations on AppWorld and OfficeBench demonstrate substantial performance gains across models ranging from 1.7B to 27B parameters.The benchmarks are described as challenging, and the gains are reported for models spanning this parameter range.

Appendices · A Data Statistics of ESAT-AW7 · A.1 Synthesis Throughput and Yield

ESAT-AW7 synthesis yields 12,242 validated tasks after staged filtering and rewriting, while trajectory generation retains 9,352 judge-approved trajectories from 14,361 input tasks. The appendices report task and trajectory throughput alongside failure-mode distributions.

  • A Data Statistics of ESAT-AW7: Table 6 reports task synthesis throughput for ESAT-AW7.The accompanying passage describes the number of tasks surviving each synthesis stage.
  • A Data Statistics of ESAT-AW7: Table 7 reports trajectory synthesis throughput for ESAT-AW7.The accompanying passage reports teacher-agent completion and judge-retention yields.
  • A Data Statistics of ESAT-AW7: Table 8 presents the distribution of failure modes during trajectory synthesis for ESAT-AW7.The supplied passages identify the table but do not provide its failure-mode values.
  • A.1 Synthesis Throughput and Yield: 29,676 candidate tasks produced by the generator were reduced to 14,361 (48.39%) after initial judging.The initial judge filtered out roughly 52% of generations as ill-formed, unsolvable, inconsistent, or mismatched to requested bucket attributes.
  • A.1 Synthesis Throughput and Yield: 13,805 tasks were successfully rewritten into concise, intent-level requests, while 556 tasks (1.87% of the original pool) were discarded.The discarded tasks failed during rewriting.
  • A.1 Synthesis Throughput and Yield: 12,242 tasks (41.25% of the initial pool) passed the second judge after re-validation against the original specifications.For rewritten tasks rejected by the second judge, the pipeline falls back to corresponding initial tasks; the passage truncates before stating the resulting count.
  • A.1 Synthesis Throughput and Yield: 11,445 trajectories (79.70%) were completed by the teacher agent from 14,361 input tasks.Completion means the agent terminated with a final answer regardless of correctness.
  • A.1 Synthesis Throughput and Yield: 9,352 trajectories (65.13% of the input tasks) were retained after the judge filtered completed trajectories for solution quality.These retained trajectories were used for supervised fine-tuning.

A.2 Failure Analysis

Among 5,009 discarded trajectories, simulation failures dominate, chiefly because agents issue too many simulated API calls within one step. Judge filtering and simulator-side validation explain roughly 87% of discards, while agent-side failures form a small tail.

  • Failure modes: Simulation failure is the dominant cause of discarded trajectories (45.38%), mainly due to simulator timeouts after excessive simulated API calls within one step.Other simulation failures occur when generated responses fail the required schema and semantic checks.
  • Failure modes: Unexpected App use accounts for 10.72% of failures and triggers early termination when the agent calls an API outside the task-derived expected set.
  • Failure modes: Agent model execution failures contribute 2.00%, while max solve steps exceeded and unexpected runtime errors each account for 0.06%.Execution failures arise from transient teacher-LLM serving errors such as rate limits or decoding errors.
  • Overall attribution: Judge filtering and simulator-side validation account for roughly 87% of all discarded trajectories, leaving agent-side failures as a small residual share.

A.3 Dataset Diversity and Coverage · B Data Statistics of ESAT-S52 · B.1 Synthesis Throughput and Yield

The synthetic dataset broadly covers task capabilities and AppWorld APIs, though multi-app tasks remain underrepresented. The ESAT-S52 pipeline filters large numbers of generated tasks and trajectories into a smaller high-quality training set.

  • A.3 Dataset Diversity and Coverage: Generated tasks cover difficulty levels, action types, and task-focus categories fairly uniformly, but two- and three-app tasks are significantly less frequent than single-app tasks.The skew may reflect the inherent difficulty of LLM-based task generation for multi-app tasks.
  • A.3 Dataset Diversity and Coverage: Successfully generated trajectories provide broad coverage across task difficulty levels, action types, and task-focus categories, while remaining skewed toward single-app tasks.Two- and three-app trajectories are significantly less frequent than single-app trajectories.
  • A.3 Dataset Diversity and Coverage: 339 of 340 AppWorld APIs (99.7%) appear in at least one successful trajectory, leaving only one API never invoked.The coverage ratio counts APIs appearing at least N times across thresholds N ∈{1, 2, 3, 5, 10, 20, 50}.
  • A.3 Dataset Diversity and Coverage: 323 of 340 AppWorld APIs (95%) are utilized at least five times, indicating that inverse-frequency sampling prevents concentration on a narrow API subset.The resulting coverage supports robust and generalized API-usage capabilities.
  • B.1 Synthesis Throughput and Yield: 32,009 generated tasks were reduced to 10,509 after the first judge, retaining 32.83% and discarding approximately 67% as unsolvable, improperly formatted, or bucket-mismatched.The surviving tasks then enter a rewriting phase for concise, intent-focused requests.
  • B Data Statistics of ESAT-S52: 9,171 successful reformulations emerged from the rewriting phase after the first judge retained 10,509 tasks.This represents a further reduction of 1,338 tasks before trajectory generation.
  • B.1 Synthesis Throughput and Yield: 8,020 of 10,509 input tasks produced teacher-completed trajectories, a 76.32% yield regardless of the final answers’ actual correctness.The teacher agent is counted as successful when it reaches a final answer.
  • B.1 Synthesis Throughput and Yield: 6,265 high-quality trajectories survived judge filtering, representing 59.62% of the original 10,509-task input for supervised fine-tuning.Quality filtering follows teacher-agent trajectory completion.

B.2 Failure Analysis

Among 4,244 discarded trajectories, judge rejection is the dominant failure mode and represents intended quality control rather than a pipeline error. Other failures arise from simulator constraints, unexpected API use, transient serving errors, and rare termination or runtime issues.

  • Failure type of trajectory generation: Judge rejection accounts for 41.35% of discarded trajectories, filtering end-to-end solutions deemed incorrect or low quality.The passage characterizes this as the intended quality-control mechanism rather than a pipeline error.
  • Failure type of trajectory generation: Simulator timeouts occur when agents issue too many simulated API calls or responses fail schema and consistency checks within the retry budget.These cases involve the simulator’s inability to produce an acceptable response before retries are exhausted.
  • Failure type of trajectory generation: Unexpected App use accounts for 22.05% of failures when agents call APIs outside the task’s expected set, triggering early termination.The expected API set is derived from the task.
  • Failure type of trajectory generation: Agent model execution failures account for 8.25% and stem from transient teacher LLM serving errors, including rate limits or decoding errors.The remaining failures include max solve steps exceeded (0.02%) and one unexpected runtime error (0.02%).

B.3 Dataset Diversity and Coverage … C.2 Failure Analysis

The synthesized dataset provides broad coverage across task and trajectory dimensions and nearly complete API coverage, while OfficeBench synthesis yields are limited mainly by agent execution failures and subsequent quality filtering.

  • B.3 Dataset Diversity and Coverage: Generated tasks cover task focus, action types, and difficulty levels fairly uniformly, but single-app tasks dominate at 64%.Two-app and three-app tasks comprise 26% and 11%, respectively.
  • B.3 Dataset Diversity and Coverage: Successful trajectories span difficulty, action type, and task focus categories, with hard tasks largest at 38%.Easy and medium tasks each account for 31%, while mixed, read, and write actions account for 38%, 32%, and 29%.
  • B.3 Dataset Diversity and Coverage: Successful trajectories remain skewed toward single-app tasks, which comprise 83% versus 14% for two-app and 3% for three-app trajectories.
  • B.3 Dataset Diversity and Coverage: The pipeline achieves near-complete API coverage across the environment, measured by the proportion of APIs invoked at least N times across multiple frequency thresholds.Coverage is evaluated for N ∈{1, 2, 3, 5, 10, 20, 50}, across 1,017 APIs.
  • C.1 Synthesis Throughput and Yield: OfficeBench task synthesis accepts 3,465 of 11,797 generated candidate tasks, a 29.37% yield, after filtering ill-formed, unsolvable, or inconsistent generations.The rewriting step reformulates 3,336 accepted tasks into concise intent-level requests, with only minor attrition.
  • C.1 Synthesis Throughput and Yield: The OfficeBench trajectory stage completes 1,889 of 3,465 input tasks (54.52%), and quality filtering retains 1,711 trajectories, or 49.38% of input tasks.Completion means the teacher agent terminates with a final answer regardless of correctness.
  • C.2 Failure Analysis: Agent model execution failure is the dominant discarded-trajectory failure mode, accounting for 77.42% of failures.These failures occur when the teacher agent does not produce a valid action or terminate properly; max solve steps exceeded accounts for 3.82% with a 30-step budget.
  • C Data Statistics of OfficeBench: Overall, most discarded trajectories arise from agent-side execution issues followed by quality filtering, while infrastructure and step-budget failures represent smaller shares.

C.3 Dataset Diversity and Coverage

The synthetic dataset provides broad, balanced coverage across task dimensions while emphasizing multi-app workflows. In OfficeBench, successful trajectories invoke most available APIs, with coverage remaining stable as frequency thresholds increase.

  • Task distribution: Two-app tasks form the plurality at 40%, compared with 35% for single-app and 25% for three-app tasks.This distribution reflects an emphasis on multi-app orchestration and complex cross-application workflows.
  • Trajectory distribution: Successful trajectories remain highly balanced across qualitative dimensions, with a slight tilt toward iteration (29%) and open (28%) tasks.Derivation accounts for 24% and constraint satisfaction for 19%.
  • API coverage: 70% (14 APIs) of the 20 available APIs appear in at least one successful trajectory.Coverage is 70% (14 APIs) at a threshold of 5 uses and declines slightly to 65% (13 APIs) at 10 uses.

D Implementation Details

The paper specifies model assignments and step limits for three-stage data synthesis, trains on masked full-trajectory turns, and evaluates with capped multi-sample inference.

  • Data synthesis: GLM-4.7-FP8 generates and validates tasks, while GLM-5.1-FP8 serves as teacher agent and API simulator for trajectory synthesis.Trajectory limits are 50 agent steps for AppWorld and 30 for OfficeBench; over-limit trajectories are aborted.
  • Training: Models are fine-tuned with supervised learning on full multi-turn trajectories, predicting each current turn from preceding turns while masking context-token loss.Training uses AdamW, a learning rate of 2 · 10−5, linear decay, 1% warmup, and a 32,768-token context length.
  • Inference and evaluation: Evaluation uses vLLM with temperature 1.0, caps trajectories at 50 AppWorld or 30 OfficeBench steps, and averages pass@1 across 8 independent samples per task.The same benchmark-specific step caps are used during evaluation as in trajectory synthesis.

E Training with AppWorld Train splits

The study tests whether duplicating filtered teacher trajectories to provide eight per AppWorld training task improves student performance. Upsampling offers no consistent benefit and often reduces accuracy relative to using valid trajectories as-is.

  • Experimental setup: Using 90 AppWorld training tasks, the teacher generates eight solution trajectories per task, after which incorrect solutions are filtered out.The number of valid trajectories therefore varies by task and is at most eight.
  • Experimental setup: The comparison evaluates unaltered filtered trajectories against randomly duplicated trajectories that balance each task at eight trajectories.The settings are labeled w/o upsampling and w/ upsampling.
  • Results: 73.14 vs. 71.21 Test-N accuracy on Qwen3.5-9B favors upsampling, whereas 51.89 vs. 53.84 Test-C accuracy favors the baseline.Upsampling slightly improves Test-N but marginally degrades Test-C for this model.
  • Results: 42.39 vs. 46.70 Qwen3.5-4B Test-C accuracy marks the largest reported gap against upsampling.The remaining three models underperform the non-upsampled baseline on both splits.

F Full Experimental Results

The section presents complete experimental results on AppWorld and OfficeBench, covering benchmark-specific metrics and task breakdowns. Results report mean performance with standard deviations across eight independent inference runs.

  • AppWorld: AppWorld results report Task Goal Completion (TGC) and Scenario Goal Completion (SGC) across Test-Normal and Test-Challenge splits.Table 16 provides performance means with standard deviations across eight runs.
  • OfficeBench: OfficeBench results provide pass-rate breakdowns for 2-app, 3-app, and overall tasks.Table 17 reports means with standard deviations across eight runs.
  • Experimental configurations: Table 16 defines AWT as AppWorld training data, S52 as 52 synthetic apps, and AW7 as 7 AppWorld apps used in data synthesis.For Qwen3.5 models, AWT added on top of ESAT did not improve performance, so ESAT-S52-AW7 results are reported for the combined configuration.

G Task Synthesis … H Simulator Details

The paper synthesizes diverse API-solvable tasks from specifications through bucketized generation, inverse-frequency coverage, judging, and intent-level rewriting. It then uses a text-only simulator to produce schema-conforming, history-consistent responses without an executable backend.

  • G Task Synthesis: Task synthesis uses API specifications to generate, judge, and rewrite a large pool of natural, multi-step tasks.The specifications contain API names, purposes, input schemas, and output schemas; GLM-4.7-FP8 performs all three roles.
  • G.1 Bucketized generation.: 360 buckets span difficulty, action type, task focus, app count, and per-app API budgets, with each bucket targeting a fixed number of accepted tasks.Buckets are processed sequentially in random order, using pinned API-count values or ranges for one, two, and three apps.
  • G.2 Inverse frequency-based sampling.: Inverse-frequency sampling prioritizes underused apps and APIs to prevent task generation from collapsing onto a narrow subset of the API ecosystem.Apps are sampled with p(a) ∝ 1/(1 + c(a)), while each selected app exposes its top-10 inverse-coverage APIs as focus APIs.
  • G.3 Task judging.: An LLM judge evaluates each candidate against the same API and bucket-specific requirements provided to the generator.The judge reasons through task solvability to assess API coverage, difficulty, and task-focus compliance.
  • G.4 Task rewriting.: A second LLM rewrites judged tasks from procedural walkthroughs into concise, intent-level user requests while preserving user-relevant details.Three in-context examples anchor the desired transformation from verbose step enumeration to compact goal-oriented wording.
  • G.5 Final re-judging.: Only rewritten tasks that pass a second judging round enter the final pool, while rejected tasks retain generation metadata for diagnostics.The second round addresses occasional loss of required entities or changes in task scope caused by rewriting.
  • G.6 Prompts for Task Synthesis: Prompt templates coordinate task generation, judging, and rewriting using sampled difficulty, action-type, and task-focus clauses plus examples and coverage information.The generator also receives sampled API documentation, negative examples, and under-utilized APIs; separate templates define judging and rewriting.
  • H Simulator Details: The API simulator is purely text-based, producing JSON responses that follow schemas, remain consistent with prior responses, and plausibly reflect the user and task.It operates without a real backend, database, or network call during trajectory synthesis.

H.1 Per-Task, Per-App Histories … K.4 AppWorld Integration

The simulator combines isolated per-task, per-app state, deterministic validation, schema checks, selective semantic judging, and structured retries to produce reliable API responses. The broader integration supplies diverse synthetic applications and converts their specifications into deployable AppWorld modules and task-generation workflows.

  • H.1 Per-Task, Per-App Histories: Per-app histories isolate each task’s accepted call-response pairs, cap prompt length, prevent cross-app leakage, and clear state when tasks finish.The simulator formats recent calls with API signatures, inputs, and simulated responses, while capping history at a configurable maximum.
  • H.2 Deterministic Input Validation: Deterministic validation rejects unknown or missing parameters, type mismatches, and constraint violations before invoking the LLM, producing repeatable errors.Checks mirror backend behavior, including scalar coercions, exact container types, bounds, enumerated values, and email formats.
  • H.3 Prompt Construction; H.4 Output Schema Validation; H.7 Prompt for Simulator: Each LLM simulation call uses fixed system rules and a structured user prompt containing the API, inputs, task context, history, and user profile; outputs undergo strict JSON and schema validation.Markdown fences are stripped, parse failures receive structured feedback, and responses are accepted when matching either documented success or failure schemas.
  • H.5 Simulator-judge; H.7 Prompt for Simulator: Read-only responses receive semantic judging, whereas structurally valid write responses are accepted to avoid doubling LLM costs for state-mutating calls.The judge receives the simulator’s specification, inputs, task, history, user profile, and current response context.
  • H.6 Retry Loop With Structured Feedback; H.7 Prompt for Simulator: A unified retry loop separates format and semantic attempt budgets and appends rejected outputs with precise structured feedback after each failure.The simulator and judge share the inlined simulation rules, while feedback distinguishes parsing, schema, and semantic errors.
  • I Agent; J Trajectory Filtering: AppWorld and OfficeBench use distinct agent prompts and trajectory judges tailored to their interaction formats, completion signals, and task requirements.AppWorld judging checks nine requirements, while OfficeBench adapts to JSON action blobs, app switching, and side-effect-based grading.
  • K Synthetic Application Suite; K.1 Domain Selection: The synthetic suite adds 52 REST-API applications across ten functional categories, totaling 1,017 typed endpoints with constraints and structured response schemas.Domains were manually curated for real-world fidelity, functional diversity, complementary coverage, and reduced redundancy with AppWorld’s original apps.
  • K.2 API Schema Design; K.3 Schema Conversion; K.4 AppWorld Integration: Generated schemas are verified, converted into AppWorld’s canonical format with traceable canary strings, and compiled into deployable modules registered separately from the nine original apps.Task prompts require solvable, realistic, interdependent, API-grounded tasks with specified coverage and required-app constraints, followed by task judging and rewriting workflows.

L Example Agent Trajectories

The trajectory decomposes a conditional banking task into API-driven steps: discover available operations, authenticate, inspect payment cards, and use transfer history and balance checks to decide subsequent actions.

  • Task decomposition: The task asks the agent to identify the latest-expiring card before 2027, conditionally transfer $50 based on 90-day history, then add $100 if the balance exceeds $100.The sequence combines card filtering, historical lookup, and two balance-dependent state changes.
  • API discovery: The agent first queries API documentation and identifies operations for listing payment cards, showing balances, withdrawing funds, and checking bank-transfer history.These APIs directly support the task’s card-selection and conditional-transfer steps.
  • Authentication: Before accessing banking data, the agent inspects the login specification, retrieves supervisor-provided credentials, and authenticates to obtain a Bearer access token.Login requires the account email and password, and the response contains an access token and token type.
  • Card inspection: After authentication, the agent prepares to call show_payment_cards with the banking-app access token and inspect each card’s expiry year and month.The documented response includes payment_card_id, card_name, expiry_year, expiry_month, and related card fields.
Loading 2607.16900v2…