AutomationBench

1 Introduction

• Business work requires agents to coordinate actions across interconnected applications while following organizational policies and filtering irrelevant or misleading information.These coordination demands are described as among the most time-consuming aspects of knowledge work.Source: §1 · p. 2
• Existing benchmarks cover browsing, computer use, or constrained tool workflows, but do not combine cross-application coordination with autonomous discovery and business rules.The passage notes that τ 3-bench includes document-based discovery in banking but remains single-application.Source: §1 · p. 2, Table 1 · p. 2
• AutomationBench tasks require agents to discover REST endpoints, make interdependent calls, apply layered rules, and handle distractors across six business domains.Success is programmatically checked against end-state data in the simulated world.Source: §1 · p. 2
• REST APIs provide a practical near-term route to high-volume orchestration because they broaden app access without purpose-built integrations and cost less than computer use.The paper describes REST APIs as a universal standard and frequent underlying interface for MCPs and CLI tools.Source: §1 · pp. 2–3
• The benchmark is designed to measure gaps in multi-step reasoning, policy adherence, and tool discovery that current models cannot reliably overcome.The stated objective is to accelerate research on cross-application orchestration.Source: §1 · pp. 2–3

2 Dataset construction and validation

• Tasks span six business domains and draw on a pool of approximately 500 API endpoints across 47 apps.The domains reflect popular workflow types among Zapier customers.Source: §2 · p. 3
• At least half the tasks are private, with public tasks for research and private tasks reserved for leaderboard evaluation.The private set began with a similar distribution and adds hardening techniques.Source: §2 · p. 3, Table 2 · p. 3
• Tasks were synthetically generated from real-customer workflow patterns without using personally identifiable or confidential customer data.The authors iterated on realism, difficulty, and app variety, and hardened tasks that seemed too easy.Source: §2 · p. 3
• Hardening adds irrelevant data, hidden information, ambiguous locations, similar decoy names, and overriding business policies to challenge reasoning beyond familiar patterns.Private tasks also receive additional hardening to reduce overfitting to a small set of challenge types.Source: §2 · pp. 3–4
• When tasks involve subjective judgment, agents must encode decisions as structured state changes that assertions can verify alongside downstream effects.Examples include updating a status, assigning a score, or routing an item to a queue.Source: §2 · p. 4
• Tasks that require policy documents explicitly direct agents to consult them, distinguishing expected policy-following from prompt-injection compliance.These policies may override reasonable defaults or external requests.Source: §2 · p. 4
• Tasks require observable state mutations and success criteria expressed as deterministic invariants, with reasoning decisions recorded in state and a single non-interactive instruction.The criteria reject free-form success judgments and require invariant-based validation.Source: §2 · p. 4
• A separate hint mode checks solvability by offering endpoint and parameter clues without revealing actual parameter values, and the authors audited hints against policy rules.The authors also used reinforcement-learning experiments to identify reward-hacking strategies and strengthen rewards.Source: §2 · p. 5

3 Tool interface

• Agents use Search to retrieve up to five relevant API schemas by BM25 keyword search, then Execute to issue method, URL, and body requests.Finding the correct endpoint and action name is part of the task.Source: §3 · p. 5
• Execute simulates retrieval or updates against Pydantic models that define app state and preserve real-schema features such as pagination, required fields, and common 4xx errors.The environment does not mimic authentication.Source: §3 · p. 5
• Tasks allow up to 50 steps, a limit rarely reached, and permit multiple tools to run in parallel within a step.The authors may increase the limit if future models regularly hit it.Source: §3 · p. 5

4 Scoring

• The grader checks expected end states across connected apps, giving the same score to different action sequences that produce correct data.Inefficient sequences can incur higher cost despite equivalent end results.Source: §4 · p. 5
• End-state grading leaves agents free to use creative approaches such as parallel calls and batched changes while keeping app side effects central.Businesses are described as caring about those side effects.Source: §4 · p. 6
• Success requires completing the task: deterministic assertions give no partial credit, and negative assertions deter shotgun actions such as emailing everyone.The assertions use exact string matches and structural checks rather than semantic text evaluation, improving reproducibility and limiting grading bias.Source: §4 · p. 6
• Scores reflect a single run, with typical run-to-run variance within 1%; the leaderboard reports average token cost per task as its primary efficiency metric.Time is harder to compare across models and providers because of rate limiting.Source: §4 · p. 6

5 Limitations

• Synthetic tasks may lack realism or be impossible, and the benchmark’s scale makes comprehensive manual review difficult.The authors manually inspected samples to reduce these risks.Source: §5 · p. 6
• Hardening increases task difficulty, but later versions may need to address reasoning gaps that remain relevant to real business workflows.The authors say those gaps may remain once current challenges are overcome.Source: §5 · p. 6
• Regular audits have not eliminated all bugs or unrealistic complexity, and the authors plan to continue fixes and version major changes.They acknowledge there is still room for improvement.Source: §5 · p. 6

6 Leaderboard

• The leaderboard header identifies model score and cost per task as reported measures.The supplied material gives the column labels but no values.Source: §6 · p. 7
• Figure 2 presents model results with reasoning effort.The caption does not provide result values or comparisons.Source: Figure 2 · p. 7

7 Results

• All state-of-the-art models score below 10%; Opus 4.7 leads at 9.9%, while Gemini and Opus pass largely different tasks.Their passing sets have Jaccard similarity 0.17; 71% of Opus’s solved tasks are not solved by Gemini, and 70% of Gemini’s passes are missed by Opus.Source: §7 · p. 7
• Sonnet 4.6 costs more per task than Opus 4.7 at every reasoning-effort level while achieving a lower pass rate.Gemini achieves some of the highest accuracy at a fraction of the cost.Source: §7 · p. 7
• Opus averages 12.6 steps and 29.8 tool calls per task, versus Gemini’s 21.8 steps and 35.4 calls.GPT 5.4 high uses 15.4 steps and 43.9 calls.Source: §7 · pp. 7–8
• Gemini 3.1 Pro passes 9.6% with API/default tools, 12.8% with Zapier, and 14.3% with Limited Zapier; Haiku 4.5 scores 1.5%, 2.0%, and 3.8%, respectively.Some models benefit disproportionately from a narrower tool surface.Source: §7 · p. 8
• The simple-domain harness baseline yields 97% for Haiku, but is excluded from the benchmark score.The authors report that models do well on simple tasks with explicit instructions.Source: §7 · p. 8
• The toolsets and public tasks are available through the public repository and Prime Intellect’s Environments Hub.Source: §7 · p. 8
• Multiple failure modes and layers of complexity require progress across several reasoning capabilities rather than one improvement.The authors argue this makes the benchmark more resilient and relevant as models improve.Source: §7 · p. 8

Common failure modes

• Models often claim success despite failure; false confidence occurs in 72% of Opus’s, 91% of Gemini’s, and 84% of GPT 5.4’s failures.Other recurring failures include incomplete searches, skipped items, and failure to follow exact instructions.Source: Common failure modes · p. 8

9 Disclaimer

• App behavior is simulated and does not exactly represent production, while all task data is fictional.The schemas are based on public documentation and used for research and evaluation; the benchmark is not affiliated with the named companies.Source: §9 · p. 9

• Example task data are simplified and include additional illustrative notes.Source: §11 · p. 10

Prompt

• The scheduling task requires using a spreadsheet policy to choose the winner, rescheduling the losing meeting, and notifying Slack with both IDs.The email-processing task requires finding a valid fire-drill message, creating and tagging an Asana task in February, and notifying Slack.Source: Prompt · pp. 10, 13

Relevant API Endpoints

• The API lists span Gmail, Sheets, Zoom, Calendar, Asana, and Slack endpoints for discovery, policy lookup, updates, and notification.The task examples require sequential calls, including adding the Asana task to a section and applying its tag.Source: Relevant API Endpoints · p. 10, Prompt · p. 13

Environment Setup

• Seeded records include conflicting and decoy meetings, unread and invalid emails, policy rows, and designated Slack channels and Asana sections.For the meeting task, a CTO attendee outranks external registrants; the email task excludes DRAFT and SUPERSEDED messages.Source: Environment Setup · pp. 11, 14

What Makes It Hard

• The meeting example tests policy over intuition, correct time matching, rejecting an external override, avoiding a timezone decoy, and choosing the right Slack channel.Calendar wins because its event has a CTO attendee, while the Zoom meeting has only external registrants.Source: What Makes It Hard · p. 12

Assertions (11 checks)

• Assertions verify the meeting task changes only the losing Zoom topic and leaves the winning Calendar event unchanged.The checks also require the Slack summary in #ops-updates to include both meeting IDs and prohibit a post to #general.Source: Assertions (11 checks) · pp. 12–13

What Makes It Hard

• The email example requires selecting the latest unread valid facilities fire-drill email, despite newer messages disqualified by policy or sender.The valid message is Jan 28 at 09:15; it specifies a February 18 due date and the Compliance tag.Source: What Makes It Hard · p. 14

Assertions (11 checks)

• Assertions verify the Asana task's name, due date, February placement, and Compliance tag, while excluding invalid-email tasks and wrong channels.The Slack notification must include the task name and due date and go only to #ops-updates.Source: Assertions (11 checks) · p. 15