How Well Does Agent Development Reflect Real-World Work?

Approach

• The authors map benchmark tasks to O*NET work domains and skills using LLM annotation with manual quality verification.Because tasks may not specify all activities needed to complete them, mapping includes disentangling composite activities and aligning informal descriptions with occupational labels.Source: §1 · p. 2

Findings

• Computer and mathematical work dominates agent benchmarks despite representing only 7.6% of U.S. employment, while management, legal, and widely prevalent interpersonal work are underrepresented.The study spans 43 benchmarks, 72,342 task instances, and 1,016 U.S. occupations.Source: §1 · p. 2

Autonomy analysis

• Agent autonomy is measured as a performance frontier across tasks of increasing complexity, enabling comparisons by domain, skill, framework, and backbone model.The authors use the measure to provide guidance on selecting autonomy levels for agent interactions across work scenarios.Source: §1 · p. 2

Benchmark design

• The authors propose benchmark-design principles of domain and skill coverage, task realism and complexity, and granular evaluation.They present these principles as ways to better capture the breadth and structure of real-world work.Source: §1 · p. 2

2.1. The Landscape of Human Work

• The domain taxonomy uses O*NET job-family and task annotations to link 23 job families, 743 occupations, and 5,806 computer-use task descriptions.The taxonomy is defined from human occupations rather than existing agent development efforts.Source: §2.1 · pp. 2–3
• The skill taxonomy defines a skill as a concrete action sequence toward a goal and expands O*NET work activities into three levels of granularity.The resulting levels contain 4, 9, and 41 nodes.Source: §2.1 · p. 3
• The human-work landscape retains physical and digital work, estimates each occupation’s digital-task share, and weights it by employment and capital statistics.Skill-level employment and capital estimates weight occupational data by O*NET activity importance; they approximate relative skill importance, not exact worker counts.Source: §2.1 · pp. 3–4

2.2. Agent Development Effort for Work

• Benchmark examples are mapped to one or more hierarchical paths in both taxonomies, capturing domain context and skills that generalize across jobs.Benchmarks are included if they operate in an interactive observation-action environment and contain tasks corresponding to real-world work activities.Source: §2.1 · p. 3, §2.2 · p. 4
• 56.5% of the domain taxonomy and 85.4% of the skill taxonomy are covered collectively; skills capture cross-occupational processes, while domains reflect occupation-specific contexts.The authors note that modality-agnostic O*NET skill categories allow digital-task benchmarks to cover skills broadly even when physical realizations are underrepresented.Source: §2.2 · p. 4
• Coverage-aware sampling adds batches of five examples until coverage grows more slowly than Δ = 0.1, aiming to preserve work diversity at lower analysis cost.Permutation tests found stable stopping size and domain and skill coverage across resampled task orderings.Source: §2.2 · pp. 4, 6
• Manual review of 90 sampled examples found 90.9% agreement for domain mapping and 89.3% for skill mapping between annotators and the LLM.The study also reports successful mapping of 91.2% of examples to domains and 95.5% to skills.Source: §2.2 · p. 6

3.1. Benchmarks Focus on Software Engineering but Miss Other Digitized Domains

• Benchmarking is concentrated in Computer and Mathematical work, which accounts for only 7.6% of U.S. employment, while large portions of the labor market remain weakly represented.General-purpose software engineering benchmarks do not fully capture domain-specific nuances in other work areas.Source: §3.1 · p. 7
• Management, Legal, and Architecture and Engineering are highly digitized—88%, 70%, and 71%—yet account for only 1.4%, 0.3%, and 0.7% of benchmark examples, respectively.The paper describes this gap as a potential opportunity for near-term productivity gains.Source: §3.1 · p. 7
• Benchmark focus also diverges from economic value: Management and low-paying, labor-intensive Personal Care and Service are both underexplored.The authors suggest that methodological convenience, including readily specified instructions and verifiable rewards, favors some domains over others.Source: §3.1 · p. 7

3.2. Benchmarks Overfocus on a Narrow Set of Skills

• Figure 3 shows that benchmark effort emphasizes Getting Information and Working with Computers, skills associated with 3.1% and 2.4% of employment, while real work draws on diverse skills.The concentration also leaves broad skill categories, especially Interacting with Others, minimally covered.Source: §3.2 · pp. 7–8, Figure 3 · p. 8

3.3. From Work Relevance to Representativeness

• Work relevance does not guarantee representativeness: many benchmarks use simplified work-related tasks that omit important context or procedural details.The paper distinguishes whether a task maps to a domain or skill from whether it captures the realistic scope and complexity of work.Source: §3.3 · p. 8
• Figure 4 shows that 77.1% of examples map to more than one domain, but only 8.5% span more than three, indicating limited cross-domain structure.Domain breadth measures the number of work domains associated with a task.Source: §3.3 · pp. 8–9, Figure 4 · p. 9
• Figure 4 shows substantial variation in skill breadth: 27.0% of tasks require one skill, while 32.6% involve four or more distinct skills.The paper treats skill breadth as a proxy for the breadth of capabilities involved, not a complete measure of procedural complexity.Source: Figure 4 · p. 9, §3.3 · p. 9

4. How Autonomous Can Agents Be?

• Autonomy is a spectrum: agents may automate narrow tasks but only augment people within more complex workflows.The section examines autonomy as a function of task complexity and offers guidance on selecting appropriate autonomy levels for deployment.Source: §4 · p. 9

4.1. Grounding Agent Autonomy in Task Complexity

• Task complexity is approximated using the number of granular workflow steps, whose semantic units abstract away repetitive low-level actions and agent-specific execution strategies.The approximation primarily captures component complexity; coordinative and dynamic complexity are difficult to estimate reliably from benchmark descriptions. Adjacent complexity levels met the relative-granularity criterion in 82.6% of sampled comparisons.Source: §4.1 · p. 10
• Autonomy is the highest task complexity an agent can complete end-to-end above a predefined success-rate threshold, with statistical confidence.The measure is intended to identify a capability boundary for selecting tasks and calibrating human oversight.Source: §4.1 · pp. 10–11

4.2. Calibrating Agent Autonomy for Real-World Tasks

• Most domains have little coverage above complexity 6, and success drops sharply with complexity in software engineering, while agents show greater autonomy on design tasks.Across skills, agents do best on self-contained mental processes and work output, but struggle with information input and interacting with others, especially on simpler tasks.Source: §4.2 · p. 11
• Framework and backbone comparisons are possible only on a few coding-focused benchmarks; OpenHands over SWE-agent and Claude over GPT show advantages, particularly at medium complexity.The trends may not hold across complexity regimes, motivating broader release of agent trajectories for more systematic and reproducible analysis.Source: §4.2 · pp. 11–12
• For a task below its target success threshold, the guidance recommends decomposing it into simpler subtasks for agent execution.The autonomy curve is also intended to expose capability boundaries that developers can target when crafting training tasks.Source: §4.2 · p. 12

5.1. Related Work

• Prior agent profiles often focus on stylistic traits, specific domains, or architecture, while autonomy remains vaguely defined and measured.The paper proposes a quantifiable autonomy measure with practical usage guidance.Source: §5.1 · p. 12
• The paper identifies a lack of systematic understanding of how existing benchmarks align with the distribution of real work.Prior work includes benchmarks for human work, workflow-integration studies, and surveys of industry practices and needs.Source: §5.1 · p. 12

5.2. Principles to Benchmark Agents for Work

• Benchmark design should broaden domain and skill coverage, including underrepresented digitized fields such as Management and Legal and skills such as Interacting with Others.The discussion recommends balancing skill categories rather than concentrating on granular activities such as Getting Information and Working with Computers.Source: §5.2 · pp. 12–13
• Benchmarks should use realistic domain and skill compositions; human-annotated tasks are contrasted with automatically synthesized tasks that may capture simplified fragments of work.When synthesis is needed for scalability, the discussion recommends grounding task generation in realistic domain and skill compositions.Source: §5.2 · p. 13
• Granular evaluation can reveal agent differences across task complexity; human demonstrations can induce workflows and automatically produced intermediate checkpoints.Benchmarks with intermediate checkpoints are more informative but costly to scale.Source: §5.2 · p. 13

6. Conclusion

• The study finds systematic mismatches between agent-development concentration and the distribution of real-world labor and economic value, and offers guidance for benchmark designers, builders, and users.The authors aim to motivate benchmarking and development that better capture the diversity, complexity, and societal importance of work.Source: §6 · p. 13

A.1. Representing Domains using O*NET

• The domain taxonomy uses O*NET’s official job-family structure to align its work domains with established occupational classifications.The paper contrasts this choice with Handa et al. (2025), which synthesizes a task hierarchy with limited human supervision and representativeness, and Patwardhan et al. (2025), which adopts U.S. Bureau of Labor Statistics sector annotations.Source: §A.1 · p. 20

A.2. Representing Skills using O*NET

• The paper uses O*NET Work Activities as its skill data because O*NET’s separate skill annotations are more conceptual and less grounded in tangible work contexts.The authors use agent-development terminology while drawing the data from work activities.Source: §A.2 · p. 20

A.3. Agentic Benchmark Selection

• Agentic benchmarks evaluate multi-step tasks requiring planning, tool use, and adaptive decisions by whether the full task is completed.Traditional benchmarks typically test single-step tasks with fixed procedures and predefined-output criteria.Source: §A.3 · p. 21
• Integrating traditional low-autonomy, level-1 tasks with agent-centric evaluations could support agents that generalize across broader work.Many underrepresented work domains already have strong coverage of low-level subtasks in traditional benchmarks, but these resources are rarely integrated into agent evaluation.Source: §A.3 · p. 21
• Robotics benchmarks offer some physical-work coverage, but motor primitives often map poorly to occupational tasks, and visual goal specifications complicate consistent taxonomy-based mapping.The study incorporated two representative datasets with natural-language descriptions and multi-step workflows.Source: §A.3 · p. 21
• Physical-work representation remains limited because robotics instructions can be templated and simple, even when provided in natural language.The paper interprets its physical-domain findings with caution and identifies richer, work-representative robotics benchmarks as a future research direction.Source: §A.3 · p. 21

• Claude Sonnet 3.5 is used to annotate whether a task requirement primarily calls for digital or physical work.Source: §A.4 · p. 22
• The classifier takes an occupational task description and returns either DIGITAL or PHYSICAL.Source: §A.4 · p. 22
• The requested output also includes a one-sentence justification for the classification.Source: §A.4 · p. 22

• 92% of predicted domain paths and 93% of skill paths aligned with human judgment in the manually verified sample.Source: §B.1 · p. 22
• Three graduate student annotators reviewed 20 randomly sampled examples per benchmark for whether mapped domains and skills captured each task’s core intent.The annotators were familiar with the O*NET taxonomy and the study’s annotation guidelines.Source: §B.1 · p. 22
• Disagreements were mostly minor and arose from under-specified tasks, which could plausibly map to different domains depending on intended execution.The passage gives building a music recommendation system as an example and reports disagreements primarily in ColBench and GAIA.Source: §B.1 · p. 22

B.2. Permutation-Based Sensitivity Analysis

• For each benchmark, the authors permuted sampled-task order 500 times and recorded tasks selected and domain and skill coverage at stopping.Source: §B.2 · p. 22, Table 2 · p. 23
• Coverage at stopping closely matched the sampled subset’s maximum, and stopping was stable across permutations, with narrow 95% confidence intervals.The authors report that the rule was not strongly sensitive to task ordering and did not prematurely terminate within the sampled pool.Source: §B.2 · p. 22, Table 2 · p. 23

C. Measuring Agent Autonomy

• Table 4 provides exemplar task instructions at varied complexity levels to help illustrate agent autonomy.Source: C. Measuring Agent Autonomy · p. 24, Table 4 · p. 24