Source-linked AI summary
APEX-Agents
Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman, Marco Burstein, Julien Benchek, David Ostrofsky, Anirudh Ravichandran, Debnil Sur, Neel Venugopal, Alannah Hsia, Isaac Robinson, Calix Huang, Olivia Varones, Daniyal Khan, Michael Haines, Austin Bridges, Jesse Boyle, Koby Twist, Zach Richards, Chirag Mahapatra, Brendan Foody, Osvald Nitski
TL;DR
AI agents lack realistic evaluation evidence for complex professional-services work. APEX–Agents introduces a 480-task benchmark requiring long-horizon, cross-application execution in expert-created environments. Gemini 3 Flash leads the leaderboard at 24.0% Pass@1, while the conclusion reports substantial remaining headroom and inconsistent performance.
Problem
Existing agentic evaluations are often narrow, contrived, and simple, limiting evidence about agents’ ability to assist professionals in real-world work.
Method
APEX–Agents evaluates eight agents on expert-created professional-services tasks using realistic worlds, files, applications, and rubric-based Pass@1 scoring.
Results
24.0% Pass@1 is achieved by Gemini 3 Flash, the highest score among the evaluated agents; the top-performing agents all score under 25%.
Takeaways & Limitations
Agents can execute complex professional-services work, but their performance remains substantially below reliable completion and varies across runs.
Takeaways & Limitations
Using Gemini 3 Flash as both judge and leaderboard model creates a risk of self-preference, although the judge does not view trajectories.
Abstract
from arXiv · showhide
We introduce the AI Productivity Index for Agents (APEX-Agents), a benchmark for assessing whether AI agents can execute long-horizon, cross-application tasks created by investment banking analysts, management consultants, and corporate lawyers. APEX-Agents requires agents to navigate realistic work environments with files and tools. We test eight agents for the leaderboard using Pass@1. Gemini 3 Flash (Thinking=High) achieves the highest score of 24.0%, followed by GPT-5.2 (Thinking=High), Claude Opus 4.5 (Thinking=High), and Gemini 3 Pro (Thinking=High). We open source the APEX-Agents benchmark (n=480) with all prompts, rubrics, gold outputs, files, and metadata. We also open source Archipelago, our infrastructure for agent execution and evaluation.
1 Introduction
APEX–Agents addresses the gap between contrived agentic evaluations and complex professional-services work by benchmarking long-horizon, cross-application tasks. It contains 480 tasks across 33 worlds, with Gemini 3 Flash achieving the highest Pass@1 score at 24.0%.
- Existing agentic evaluations often use narrow, contrived, simple tasks that provide limited evidence about real-world professional assistance.
- APEX–Agents evaluates whether agents can execute complex professional-services work requiring advanced knowledge, multiple applications, reasoning, and long-horizon planning.
- The benchmark was built from data-rich project worlds in which professionals planned work, researched, and produced customer-ready deliverables before creating agent tasks.Agents receive access to each world’s data and software, while tasks take experienced professionals an estimated 1–2 hours on average.
- 480 tasks span 33 worlds, and tasks are evenly divided across investment banking, management consulting, and corporate law.The benchmark data and Archipelago execution and evaluation infrastructure are open source.
- 24.0% is Gemini 3 Flash’s highest Pass@1 score, followed by GPT-5.2 at 23.0%; the difference is not statistically significant.Claude Opus 4.5 and Gemini 3 Pro each score 18.4%, while both open-source models score under 5%.
2 APEX Survey
The APEX Survey examines how elite professionals spend their working time to inform benchmark construction. It surveyed 227 participants across finance, consulting, and law, and found that core activities comprise 47% of total time.
- The survey covered 227 professionals: 58 financial and investment analysts, 77 management consultants, and 92 lawyers.Participants averaged 10.8 years of professional experience.
- Core activities accounted for 47% of participants’ total working time.Participants also reported time spent on learning, administration, communications, meetings, and non-productive activities.
- Survey responses were manually reviewed and grouped into 18 inductively identified categories of core work.
3 Benchmark design and dataset
APEX–Agents constructs realistic professional-services environments from expert-created project worlds, files, applications, tasks, rubrics, and gold outputs. The benchmark contains 480 reviewed tasks distributed across 33 worlds and three occupations.
- Worlds and applications: Each world begins with an expert-created project scenario and assigns role-appropriate responsibilities involving emails, research, deliverables, and iterative work products.
- Worlds and applications: The 33 worlds include 10 investment-banking, 11 management-consulting, and 12 law worlds, averaging 166 files and nine applications with 63 tools each.
- Tasks and rubrics: Tasks are long-horizon, single-turn prompts that require using in-world files and applications, with 8–20 tasks per world and 14.5 tasks on average.
- Tasks and rubrics: 422 of 480 tasks require a console message, while the remainder require creating or editing spreadsheets, documents, or presentations.
- Tasks and rubrics: Rubrics contain independently gradable criteria and grading targets, with 1–10 criteria per task and a mean of 4.06.Experts also created gold outputs in the requested format and tagged prompts and rubrics with workflow and expected-time metadata.
- Baselining study: 20% of tasks were independently executed by experts; sampled tasks took 1.37 true hours versus a 1.70-hour estimate, a 33% overestimate.The independent executions also identified minor prompt, rubric, or metadata problems in 10% of tasks.
4 Evaluation
Evaluation runs agents repeatedly, grades rubric criteria with a judge model, and reports task-level Pass@1. The judge performs strongly against human labels, but self-preference and small score differences require caution.
- Execution and grading: Each agent executes every task eight times, producing 30,720 trajectories, with trajectories exceeding 250 steps marked as failures.The step limit reflects observed looping behavior beyond that point.
- Execution and grading: A judge model grades each rubric criterion as met or not met using the prompt, output, action-change log, and criterion, but not the trajectory.
- Judge validation: 98.5% accuracy, 96.7% precision, 98.1% recall, and 97.4% positive-class F1 were achieved against 747 human-labeled criteria.Differences below 1 percentage point should be interpreted cautiously.
- Judge validation: Gemini 3 Flash is both the judge and a leaderboard model, creating a self-preference risk that is mitigated because the judge does not view trajectories.For Gemini 3 Flash, the ground-truth evaluation found one false positive, or a 1.2% false-positive rate.
- Metrics: Pass@1 is the proportion of tasks whose outputs meet all criteria, answering the probability that one run on a uniformly selected task passes.
- Metrics: Pass@8 measures passing at least once across eight attempts, whereas Pass^8 measures passing on every attempt and therefore assesses consistency.The evaluation also reports mean criteria passed per task and estimates Pass^k for k < 8.
5 Results on APEX–Agents
APEX–Agents results show substantial but inconsistent agent capability: leaderboard Pass@1 remains below 25%, while repeated runs and partial-credit measures are higher. Performance also varies by job, output type, tool behavior, and failure mode.
- Pass@1: 24.0% is the highest Pass@1 score, achieved by Gemini 3 Flash; GPT-5.2 scores 23.0%, while Claude Opus 4.5 and Gemini 3 Pro each score 18.4%.The two best agents’ difference is not statistically significant, but both outperform the other models.
- Performance by job: 27.3% is the top Pass@1 for investment banking, 22.7% for management consulting, and 25.9% for corporate law.GPT-5 and GPT-5.2 tie in investment banking, GPT-5.2 leads consulting, and Gemini 3 Flash leads corporate law.
- Repeated runs: 40.0% is the highest Pass@8 score, compared with 13.4% for the highest Pass^8 score, indicating capability that is not consistently reproduced across runs.GPT-5.2 leads Pass@8, while Gemini 3 Flash leads Pass^8.
- Mean score: 39.5% is Gemini 3 Flash’s highest mean score, followed by GPT-5.2 at 38.7% and Claude Opus 4.5 at 34.8%.Mean scores are much higher than Pass@1, and partial-credit outputs contribute to this difference.
- Failure analysis: 36 trajectories contain unwanted file deletions, representing 0.12% of all trajectories; GPT-5.2 accounts for 21 of them.Deletions were never requested in the task prompts.
- Failure analysis: Successful trajectories use 0.22 more unique tools but 5.95 fewer steps and 5.66 fewer tool calls than paired failing trajectories.They are also 3 percentage points more likely to use code at least once, consistent with avoiding repeated unproductive tool use.
6 Conclusion
APEX–Agents finds that frontier agents can perform complex professional-services work, but their success is limited and inconsistent. The benchmark and Archipelago infrastructure are released openly to support further research and expansion.
- Conclusion: All top-performing agents score under 25% on Pass@1 and no more than 40% on Pass@8 or mean score.Failures include timeouts, zero scores, and partial credit, with substantial variance across runs.
- Conclusion: The benchmark demonstrates complex-task capability alongside inconsistency, shown by variance across runs and the drop from Pass@8 to Pass^8.Partial-credit outcomes indicate that some failed trajectories still produce useful output.
- Open research: APEX–Agents and Archipelago are open-source, with planned future expansion in task horizon, world complexity, and benchmark value.Archipelago supports running and evaluating agents, while the benchmark release includes the research dataset and infrastructure.
A Archipelago
Archipelago is the open-source infrastructure used to run and evaluate agents in APEX–Agents. Its design combines containerized environments, agent runners, and grading based on changes to the simulated world.
- Architecture: Archipelago has three components: an Environment, an Agents runner, and a Grading system.All three components are Docker containers deployable on orchestration platforms such as Kubernetes or Modal.
- Environment: The Environment exposes multiple applications through a unified Model Context Protocol gateway inside a containerized sandbox.This provides the multi-application setting used for agent execution.
- Agents runner: The Agents runner executes language models with different agentic harnesses; APEX–Agents uses a ReAct toolbelt.The runner separates model execution from the environment and grading components.
- Grading: The Grading system evaluates outputs by comparing before-and-after world snapshots using verifiers.This supports evaluation of changes made across the applications in an agent’s environment.
- Work grounding: The benchmark’s workflows are grounded in professional work data from the APEX Survey, whose 227 participants reported time allocations and core-activity categories.Tables summarize workflow frequencies, professional time use, and the five most reported core-work categories for each job.
D.1 ReAct toolbelt
The ReAct toolbelt interleaves reasoning and action in a loop, with explicit planning, tool execution, and intentional finalization. Context-management choices limit tool overload and token exhaustion.
- ReAct loop: At each step, the agent observes the state, reasons about the next action, executes tools, and repeats.This implements the ReAct paradigm as an interleaved reasoning-and-acting loop.
- Termination: An explicit final_answer tool call is required to complete a task with structured answer and status output.The requirement enforces intentional termination rather than relying on implicit stopping.
- Context management: Agents initially receive meta-tools for discovering and managing tools plus task-planning capabilities, while additional tools are exposed as needed.This toolbelt design is intended to prevent tools from bloating the model context.
- Agent instructions: All agents use the same system prompt and must attempt tasks independently without asking for help.The prompt also directs agents to use code execution when needed and to calculate exact values unless rounding is specified.
- Filesystem assumptions: Relevant files are assumed to be under /, except code execution must explicitly use /filesystem/.This establishes different filesystem conventions for ordinary tools and code execution.
E Model details
The appendix identifies the model configurations evaluated on APEX–Agents and reports pairwise statistical comparisons among them.
- Table 9 lists the configurations of models tested against the APEX–Agents benchmark.
- Table 10 reports pairwise Pass@1 scores for the tested models using McNemar’s exact test with Benjamini–Hochberg correction.
- The pairwise analysis applies a Benjamini–Hochberg correction across 28 comparisons.
G Tools most frequently used by agents
Agents most often used tools for code execution, tool management, file browsing, spreadsheet reading, and document or PDF analysis.
- 256,000 uses made Code execution the most frequently used tool, followed by Add tool to the toolbelt at 200,000.
- 164,000 uses involved listing files, while spreadsheet access included 127,000 Read spreadsheet tab uses and 42,000 List tabs in a spreadsheet uses.
- PDF and document tools were also frequent, including Search the PDF at 86,000, Read PDF at 55,000, and Read document content at 45,000.
- Agents used Inspect tool with the toolbelt 78,000 times and Read an image 37,000 times.