Source-linked AI summary
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, Graham Neubig
TL;DR
The paper addresses limited objective evidence about how well LLM agents perform diverse workplace tasks and where they fall short. It introduces TheAgentCompany, a simulated software-company benchmark involving browsing, coding, programs, and coworker communication, and finds that agents can autonomously perform varied work tasks but are not close to automating every evaluated task.
Problem
Objective benchmarks are needed to assess LLM agents across workplace tasks while documenting tasks they cannot perform, given the implications for industry and economic policy.
Method
TheAgentCompany evaluates AI agents in a self-contained simulated software company using tasks drawn from workplace categories and requiring web browsing, coding, program execution, and coworker interaction.
Results
Agents can autonomously perform a wide variety of everyday workplace tasks, but they are not close to automating every task in the benchmark, including well-scoped administrative and coding work.
Takeaways & Limitations
The results provide a nuanced picture: simpler workplace tasks may be solvable autonomously, while more difficult long-horizon tasks remain beyond current systems.
Takeaways & Limitations
The benchmark emphasizes straightforward tasks with automatically evaluable goals, covers only selected tasks within occupations, uses two baseline agent scaffolds, and does not compare agents with human professionals.
Abstract
from arXiv · showhide
We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at accelerating or even autonomously performing work-related tasks? The answer to this question has important implications both for industry looking to adopt AI into their workflows and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that the most competitive agent can complete 30% of tasks autonomously. This paints a nuanced picture on task automation with LM agents--in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. We release code, data, environment, and experiments on https://the-agent-company.com.
1 Introduction
TheAgentCompany addresses the lack of objective benchmarks for measuring AI agents on consequential workplace tasks by evaluating them in a simulated software company. Its experiments show that agents can autonomously complete some everyday work, but not all tasks.
- Motivation: The paper motivates objective benchmarks that measure both workplace-task acceleration and the tasks agents cannot perform.The authors frame this gap as important for understanding the implications of AI assistance and automation in work-related settings.
- Benchmark: TheAgentCompany evaluates agents on software engineering, project management, financial analysis, and other workplace tasks requiring browsing, coding, and coworker interaction.The environment is open-source and self-hostable, with evaluators that assign partial credit for partially correct answers.
- Results: 30.3% of tasks were autonomously performed by the best-performing model, Gemini 2.5 Pro.The supplied experiment passage reports this result but is truncated after the percentage.
- Results: Agents can autonomously perform a wide variety of everyday work tasks, but they are not close to automating every task in the benchmark.The remaining tasks include well-scoped administrative and coding work in a software company’s day-to-day setting.
2 Benchmark Desiderata and Comparison to Other Benchmarks
TheAgentCompany is designed to evaluate workplace-relevant agent capabilities across diverse tasks, communication, long-horizon execution, interfaces, and reproducible infrastructure. These desiderata distinguish it from benchmarks with narrower task or interaction coverage.
- Coverage of Multiple Work-related Tasks: The benchmark includes diverse, realistic, professional tasks typically completed by multiple roles in a software engineering company.This broader coverage is intended to support statements about AI agents across multiple categories of real-world work.
- Requirement for Interaction: Agents communicate with simulated coworkers to request and provide information during complex workplace tasks.Most other benchmarks do not measure this kind of workplace communication, while τ-bench focuses on customer-service interaction.
- Long-horizon Tasks with Checkpoints: The benchmark contains longer tasks requiring more consecutive work and uses granular checkpoints to measure subtasks within larger goals.This design targets real-world tasks that require many steps and provides progress-sensitive evaluation.
- Versatile Environment Interface: TheAgentCompany supports web interfaces, programs, command-line terminals, and communication tools rather than focusing on only one or two interfaces.This interface coverage is intended to support a diversity of workplace tasks.
- Self-hosted and Reproducible: The benchmark is fully self-hosted and reproducible for comparisons that remain constant over time.This contrasts with benchmarks lacking execution environments or requiring third-party hosted platforms.
3 TheAgentCompany Environment Setup
TheAgentCompany recreates a software startup through a local sandbox, internal websites, reproducible data, and simulated colleague communication. This environment lets agents perform work across code, documents, task management, and messaging systems.
- Company Setting: The benchmark is set in an imaginary software engineering startup whose tasks are inspired by work performed inside such companies.Its environment combines local and intranet components.
- Local Workspace: The local workspace is a sandboxed Docker environment where agents use a browser, code editor, and Linux terminal.The workspace runs locally on the agent’s host and is designed to avoid affecting other parts of the evaluation machine.
- Intranet: The intranet hosts open-source, self-hostable websites for code, documents, project management, and communications.These components follow a reproducible, self-contained environment design.
- Intranet: GitLab hosts code repositories and wikis, OwnCloud supports file storage and collaborative editing, Plane manages tasks and roadmaps, and RocketChat supports internal messaging.Together, these websites represent common software-company work systems.
- Reproducibility: The websites are reproducible and resettable with mock data combining real-world software-project data and manually curated corporate-role data.The environment also provides a pre-packaged virtual-machine operating-system image.
- Simulated Colleague Communication: Agents use RocketChat to message simulated colleagues and obtain information unavailable in the original task description.The colleagues are generated through Sotopia and receive profiles describing their roles, responsibilities, and project affiliations.
4 Task Structure
Tasks combine natural-language intents, intermediate checkpoints, programmatic or LLM-based evaluators, and execution workflows. The benchmark measures both completion capability and operational efficiency while rewarding partial progress and full completion.
- Task Structure: Each task contains an intent, checkpoints, evaluators, and code for initializing and finalizing the environment.The workflow comprises initialization, execution, and finalization stages.
- Task Intent: Task intents are English descriptions designed to let a human worker complete the task without asking the user for further instructions.Workers may still need to ask simulated coworkers questions.
- Checkpoints: Checkpoints represent intermediate milestones with point values measuring progress toward task completion.They can assess action completion, data accuracy, or collaboration.
- Evaluators: Evaluators inspect environment states or agent trajectories to determine whether checkpoints were completed.Most evaluators are deterministic Python functions, while complex and subjective deliverables may use LLM-based evaluation with rubrics or reference outputs.
- Evaluation Metrics: S_full equals 1 when all checkpoints pass and 0 otherwise, making full task completion a binary metric.The metric evaluates whether the agent successfully completed the entire task.
- Evaluation Metrics: S_partial = 0.5 · Result/Total + 0.5 · S_full combines fractional checkpoint progress with a binary full-completion bonus.This formulation gives partial credit proportionally while reserving half the score for passing all checkpoints.
- Efficiency Metrics: Number of steps counts LLM calls, while cost per instance estimates API expense from prompt and completion token usage.These metrics quantify operational effort and monetary cost.
5 Task Creation
The benchmark tasks were selected to cover diverse, work-motivated activities within a software-company setting, using occupational data and expert input while prioritizing concrete goals and success criteria. The authors caution that the selected tasks are not an exhaustive sample of occupational work.
- 5.1 Choosing Task Categories: TheAgentCompany uses O*NET data to identify populous, high-value occupations and select a software-company setting suitable for digital tasks.The selection also excludes occupations requiring extensive physical labor.
- 5.2 Choosing Tasks: Tasks draw on O*NET task lists, coauthor expertise, and language-model brainstorming to provide diversity within the chosen setting.
- 5.2 Choosing Tasks: The task set emphasizes concrete activities with well-defined goals and success criteria rather than exhaustive occupational coverage.
- 5 Task Creation: Task construction involved specifying intent, checkpoints, evaluation criteria, required data, and scripts to configure the environment.
- 5 Task Creation: 20 contributors spent over two months and approximately 3,000 person-hours creating, testing, and verifying the tasks.Some complex tasks required more than 10 hours each to design, implement, test, and verify.
6 Baseline Agents
The evaluation uses agents that can browse, operate a terminal, and write and execute programs, including OpenHands CodeAct Agent with Browsing and OWL-RolePlay.
- 6 Baseline Agents: The baseline agents must browse the Web, operate a local workspace through a terminal, and write and execute programs.
- 6 Baseline Agents: The study adopts OpenHands’ CodeAct Agent with Browsing and OWL-RolePlay, a multi-agent framework for real-world task automation.
7 Experimental Results
Experiments show substantial variation across models, agent designs, platforms, and task categories. Agents perform relatively better on some software-engineering tasks but struggle with social interaction, complex interfaces, and many administrative or financial activities.
- 7.1 Result Overview: GPT-4o yields 8.6% success with OpenHands versus 4.0% with OWL RolePlay, reflecting differences in technical design.OpenHands better preserves consistency over long-horizon tasks, while OWL RolePlay can lose progress during delegated browsing.
- 7.1 Result Overview: Llama 3.1 (405B) is the strongest open-weight model, nearly matching GPT-4o but remaining behind Gemini 2.5 Pro.
- 7.1 Result Overview: Llama 3.1 (405B) takes more steps, costs nearly 2x more, and achieves lower success than GPT-4o.The authors suggest open-weight models are not always the most cost-effective choice for highly complex tasks.
- 7.1 Result Overview: Llama 3.3 (70B) reaches a 6.9% success rate, matching the much larger Llama 3.1 (405B) model at lower cost.
- 7.2 Analysis: Most models struggle with RocketChat and ownCloud, where social interaction and complex web-based Office interfaces create major difficulties.
- 7.2 Analysis: Task-category results reveal higher performance on software-development engineering than on seemingly easier administrative and financial tasks.Failures involve document understanding, communication, complex software navigation, and repetitive processes.
- 7.2 Analysis: Overall, agents fail a majority of benchmark tasks and exhibit recurring errors involving social reasoning, Web browsing, and fabricated shortcuts.
8 Implications and Future Directions
TheAgentCompany shows that current agents can autonomously perform some realistic workplace tasks but still fail to automate most tasks, especially socially interactive, complex-interface, and private-knowledge work. The benchmark is an initial step whose scope and task design leave substantial room for expansion.
- Implications: Current state-of-the-art agents fail to solve a majority of TheAgentCompany tasks, leaving a substantial gap from automating most human workplace jobs.The benchmark focuses on relatively simplified administrative and coding tasks, so this gap appears even in a constrained setting.
- Implications: Tasks involving social interaction, complex professional user interfaces, or work performed without substantial public resources are the most challenging.
- Future Directions: The benchmark is intended as a first step toward understanding how AI may affect workplace tasks, rather than a complete account of workplace automation.
- Implications: The authors report that newer models are becoming more capable, more cost-efficient, smaller, and closer in open-weights performance to proprietary frontier models.
- Future Directions: Its future extensions include tasks from other industries, physical labor, vague goals, and longer-horizon efforts such as taking a product from conception to execution.
B Benchmark Construction
TheAgentCompany is a hand-curated, domain-expert-built benchmark designed to approximate realistic software-company work across multiple professional domains and tools. It adds simulated human interaction, realistic pitfalls, and interconnected workflows to existing benchmark settings.
- Task Curation: Every benchmark task was hand-curated by domain experts to reflect complex and realistic workflows rather than scraped from publicly available tasks.
- Task Curation: Tasks are grounded in the US Department of Labor’s O*NET database, which catalogs jobs, responsibilities, and task requirements.
- Task Coverage: The benchmark spans software development, project management, data science, administration, human resources, and finance tasks created or validated by relevant professionals.
- Realism: Simulated LLM-based colleagues introduce human interaction, unpredictability, and realistic traps or pitfalls into workplace tasks.
- Realism: Example workflows combine multiple tools and services, including issue tracking, communication, repositories, code execution, and document sharing.
C Example Tasks
Example tasks require agents to complete interdependent, multi-tool workflows and receive partial credit through checkpoint-based evaluation. The baseline environment combines browsing, terminal and Python execution, simulated colleagues, and deterministic or LLM-based evaluators.
- Example Workflow: A representative sprint-management task requires issue updates, team communication, repository operations, report generation, and feedback incorporation across multiple tools.
- Example Workflow: The agent fully completed issue reassignment and colleague notification, earning 2/2 and 1/1 points respectively.
- Example Workflow: The agent cloned the repository but failed to run code coverage, earning 1/2 points for that checkpoint; later report steps were not completed.
- Evaluation: 25% was the example agent’s final score, calculated from 4 awarded points out of 8 possible points under the partial-completion formula.
- Baseline Agent: OpenHands’ baseline agent uses browser interaction, bash commands, Python execution, and observations from terminals, programs, and browser accessibility trees.
- Evaluation: The benchmark evaluates checkpoints and final results with mostly deterministic programs, supplementing them with LLM judges for complex, subjective deliverables.
- Simulated Colleagues: Simulated colleagues and their interactions are part of the environment, with most NPC context exchanges occurring within a few hundred tokens.
I Detailed Experimental Results
The detailed results are organized by the platforms required by tasks and by the professional nature of those tasks. This breakdown supports analysis of performance across interfaces and job categories.
- Results Breakdown: Table 4 breaks down performance by required platform, while Table 5 breaks it down by task type and associated professional department.
J Limitation
The benchmark was tested by people for feasibility, but it does not include human performance data because completing tasks at scale was costly. The supplied materials also include simulated communication examples and performance-table captions without reported results.
- Human evaluation scope: Human performance data were not collected because individual tasks can take 10 minutes to several hours, making large-scale recruitment costly.People tested and completed all tasks to verify feasibility and the evaluator, but no scalable human-performance study was conducted.
- Reported evidence: The supplied performance-table captions identify percentage-based platform and task-nature analyses but report no numerical findings here.Table 4 concerns platform requirements, while Table 5 concerns task nature.
- Human evaluation scope: The absence of human performance data does not affect evaluation of the current models in the benchmark.