Source-linked AI summary
Work Agents: Systems, Capabilities, and Optimization
Zishan Xu, Enjun Du, Mingqi Gao, Cheng Yang, Junhao Shen, Zirong Chen, Zhiyuan Yao, Yifu Guo, Yuquan Lu, Jinyang Huang, Zhengxi Lu, Siyi Liu, Ziyu Zheng, Yongqi Zhang, Difan Zou, Weiwen Liu
TL;DR
Research lacks a unified account of how agents reliably accomplish delegated work across tools, applications, and workflows. The paper develops a work-centric framework for agents, their conditions, capabilities, optimization, and evaluation. It concludes that reliable digital work depends on connecting task requirements, actions, and outcomes as work unfolds.
Problem
Existing research lacks a unified perspective on how agents reliably accomplish delegated work across tools, applications, and workflows.
Source: §1 · pp. 4–5Method
The paper frames work agents as model–harness systems in stateful workspaces and organizes work conditions, forms, capabilities, optimization, and evaluation around work outcomes.
Source: §1 · p. 5Results
Reliable digital work depends on keeping task requirements, actions, and outcomes connected as work unfolds.
Source: §7 · p. 61Takeaways & Limitations
Openness makes discovery part of work, while persistence carries responsibility across stages and events.
Source: §7 · p. 61Answer-level evaluation suits well-defined outputs, but a correct answer does not establish that the operations behind it were correct.
Source: §5.2.1 · p. 44
Abstract
from arXiv · showhide
As foundation models evolve from information generation to digital work execution, agents are increasingly capable of completing complex tasks across tools, applications, and workflows. However, existing research lacks a unified perspective on how agents reliably accomplish delegated work. To address this gap, we present a work-centric, system-level perspective that views work agents as systems combining foundation models and runtime harnesses to execute tasks in stateful workspaces and deliver verifiable outcomes. We introduce two complementary dimensions, work persistence and work openness, to characterize working conditions, and classify digital work into five forms according to task scope and organization. Building on this framework, we identify five core capability families required for reliable work execution and examine approaches to model optimization, harness optimization, and their joint optimization. We further analyze existing evaluation methods and metrics from the perspective of work outcomes. Finally, we discuss future research directions, including sustained work delegation, adaptation to open environments, and the interaction between work execution and capability improvement. This paper provides a unified framework for understanding, optimizing, and evaluating work agents, supporting the development of systems capable of delivering reliable, verifiable, and practically valuable work outcomes.
1 Introduction
As agents take on complex, multi-stage work, the paper argues for a work-centric view that connects objectives, system mechanisms, workspace states, outcomes, and verification. It introduces a framework for characterizing work conditions and capabilities, examining optimization and evaluation, and identifying challenges for sustained delegation.
1 Introduction
- Work responsibilities increasingly span stages and depend on evidence in related files, records, and applications established by earlier actions.The paper focuses on digital knowledge work across office productivity, professional services, enterprise operations, software engineering, and scientific research.Source: §1 · pp. 4–5
- A work agent combines a foundation model and runtime harness to perform delegated tasks in a stateful workspace and produce outcomes checked against requirements.This shifts the unit of analysis from isolated outputs or tool calls to the process of accomplishing digital work.Source: §1 · p. 4
- The paper presents a unified work-agent framework spanning six system elements, five work forms, and the complementary conditions of persistence and openness.The six elements are task, model, harness, workspace, work outcome, and verification.Source: §1 · p. 5
- Reliable work execution requires five capability families and consideration of model, harness, and joint optimization.The paper examines how training environments, data, reward design, and external system support contribute to execution.Source: §1 · p. 5
- The paper analyzes benchmarks, metrics, judging methods, and execution environments in terms of artifact quality, state changes, completion, and constraint satisfaction.This outcome-oriented perspective is intended to clarify practical performance and reliability.Source: §1 · p. 5
- Future research directions include sustained responsibility, adaptation, outcome verification, and the interaction between work execution and capability improvement.The paper discusses personal work agents, human digital twins, and organizational work systems.Source: §1 · p. 5
2 Foundation of Work Agents
The section defines work agents as model–harness systems that act in stateful workspaces and have outcomes verified against task requirements. It distinguishes work conditions by when responsibilities close and how much of the task world agents must discover.
2.1 Definition of Work Agents
- A work agent’s six elements interact in an execution loop: the harness provides context, the model proposes actions, and workspace observations and verification feedback inform later decisions.The model and harness together form the agent; the task, workspace, outcome, and verification define the work for which it is accountable.Source: Figure 1 · p. 6, §2.1 · p. 7
- A work agent uses a foundation model and harness to pursue a delegated objective in a stateful workspace, with outcomes assessed against task requirements using observable evidence.The definition ranges from answering a question over supplied material to changing workspace state or completing a business process.Source: §2.1 · pp. 6–7
2.2 Working Conditions
- Work conditions vary along two distinct dimensions: persistence concerns when responsibility closes, while openness concerns what the agent must discover during execution.The dimensions impose different demands on understanding the environment, managing dependencies and state, and verifying outcomes.Source: §2.2 · p. 7
2.2.1 Work Persistence
- Persistence ranges from responsibility that closes at delivery to responsibility that stays open across stages and events, requiring the agent to track what remains unresolved.The distinction is separate from task difficulty and execution length: a long task can close at delivery, while a short recurring obligation can remain open for a quarter.Source: §2.2.1 · p. 8
2.2.2 Work Openness
- Openness ranges from a predefined task world to one the agent must discover, including relevant objects, information, interfaces, rules, state, and authority.Examples progress from supplied material and explicit tools to environments requiring navigation, cross-application search, or discovery of permissions and roles.Source: §2.2.2 · pp. 8–9
2.2.3 Combining Persistence and Openness
- Persistence and openness are conceptually distinct, yielding four combinations; together they describe working conditions rather than the form of work.Question answering, information seeking, service workflows, and ongoing work under changing conditions illustrate the combinations.Source: §2.2.3 · p. 9
2.3 Core Capabilities of Work Agents
Reliable work execution depends on five capabilities that connect task understanding, decisions, dependencies, workspace actions, and outcome checks. They operate as a cycle through shared work context, with persistence extending work across stages and events and openness increasing what must be discovered.
2.3 Core Capabilities of Work Agents
- The five capabilities form a cycle that passes evidence, decisions, plans, actions, observations, and results through shared work context.The context retains requirements and completion criteria, evidence and object links, plans and pending dependencies, and observed state with check status.Source: §2.3 · pp. 9–10
- Persistence extends each capability across stages and events, whereas openness increases what the agent must discover during execution.The illustrated work concerns updating a business report and related workbook and slides when new data arrive.Source: §2.3 · pp. 10–11
2.3.1 Perception and Understanding
- Perception and Understanding identifies task objects, interprets their content and structure, and traces relationships to produce source-traceable evidence and edit locations.A workbook value is usable only when its header, unit, and reporting period are known, and an edit requires locating its target cells or slide element.Source: §2.3.1 · p. 11
2.3.2 Reasoning and Decision Making
- Reasoning and Decision Making weighs goals, constraints, evidence, and domain knowledge to choose an approach, while recording the decision basis for later checks.Further constraints may need to be established during execution, and revisions can require gathering more information before acting.Source: §2.3.2 · p. 12
2.3.3 Planning and Coordination
- Planning and Coordination specifies subgoals, inputs, dependencies, ordering, and participant handoffs so dependent actions and contributions fit together.Plans can include pending dependencies and must account for when collaborator information or authorization is needed.Source: §2.3.3 · pp. 12–13
2.3.4 Execution and State Management
- Execution and State Management acts on workspace objects through tools or interfaces and records progress, state changes, results, and pending items.Attempted actions, accepted calls, committed effects, and verified effects are distinct; an uncertain effect remains pending until inspected.Source: §2.3.4 · p. 13
2.3.5 Verification, Recovery, and Replanning
- Verification, Recovery, and Replanning checks results against requirements, repairs local defects, and revises remaining steps when prerequisites or conditions change.A changed prerequisite requires checking downstream results again, while uncertain effects should be inspected before retaining, repairing, or retrying an operation.Source: §2.3.5 · pp. 13–14
2.4 Work Agents and Related Agent Paradigms
The work-agent perspective organizes related agent research around delegated objectives and outcomes checked against task requirements. It relates long-horizon, computer-use, and office-agent work as complementary concerns that may need to be combined within one work process.
2.4 Work Agents and Related Agent Paradigms
- Work agents are defined by delegated objectives and requirement-checked outcomes, rather than by execution duration, interface, or artifact type.The framework places related paradigms within a work process from task requirements through workspace execution to verified outcomes.Source: §2.4 · p. 14, §2.4.4 · p. 15
2.4.1 Long-Horizon Agents
- Long-horizon agents study progress, context, and state over extended execution, while work persistence asks when responsibility may close and requires tracking open obligations and re-verifying earlier results.Persistent delegation includes responding to participants and later events and checking prior results against the current workspace.Source: §2.4.1 · pp. 14–15
2.4.2 Computer-Use Agents
- Computer-use agents ground task intent in executable interface operations, while the work-agent view asks how those operations contribute to the delegated outcome and its dependencies.The broader view also considers responsibility closure, permissions, and dependencies that may not be visible through the interface.Source: §2.4.2 · p. 15
2.4.3 Office Agents
- Office agents focus on creating and modifying work artifacts, while the work-agent framework analyzes office outcomes alongside software, service, and organizational work.A delegated task can encompass an updated workbook, related business-record changes, and approval required before publication.Source: §2.4.3 · p. 15
2.4.4 Relationships among Paradigms
- Long-horizon, computer-use, and office-agent paradigms track different parts of a reporting task, and a single system may need all three within one work process.The comparison uses updating a business report when new data arrive and relates research to the capabilities that must be combined for a delegated task.Source: §2.4.4 · p. 15, Table 2 · p. 16
3 Work Forms: What Work Do Agents Undertake?
The five work forms organize delegated digital work by the scope of the outcome and the dependencies the agent must manage. As scope expands from information or local changes to workflows and workspaces, more intermediate results, handoffs, and object relationships become part of completion.
3 Work Forms: What Work Do Agents Undertake?
- The five forms progress from informational responses and local actions to dependent multi-step objectives, procedural workflows, and workspace-wide discovery and coordination.As scope expands, agents manage more dependencies around outcomes, including intermediate results, stage transitions, and relationships among work objects.Source: §3 · p. 17
3.1 Conversation Level
- Conversation-level work produces information from a request and evidence; its completion does not require changing source artifacts or external work-object states.Response complexity depends on interpreting material such as prose, tables, images, charts, and linked passages, then applying the requested inference.Source: §3.1 · p. 17, §3.1.2 · pp. 17, 19
3.2 Atomic Action Level
- At the Atomic Action Level, the agent changes a clearly targeted work object while preserving properties the request leaves unchanged.The level reflects the local scope of the objective, not the number of mouse, keyboard, or API operations needed to execute it.Source: §3.2 · p. 19
3.3 Multi-step Level
- Multi-step work coordinates dependent operations, adapts later actions to intermediate results and feedback, and is judged by the final objective.Intermediate results can provide inputs for later decisions, evidence for an answer, or state needed for an operational change.Source: §3.3 · p. 20, §3.3.1 · p. 20, §3.3.2 · p. 20
3.4 Workflow Level
- Workflow-level delegation includes the process around an objective, so completion requires both the final output and required procedural conditions or handoffs.A correct forecast cannot take effect while a required approval is missing; CXBench scores final outcomes and process compliance separately [131].Source: §3.4 · p. 21, §3.4.2 · p. 21, §3.4.1 · p. 21
- Stages created by an agent system are not themselves delegated workflow stages unless the requester’s result depends on them.A research and citation sequence can organize execution without being part of the request when any equally well-sourced report would satisfy it.Source: §3.4 · p. 21
3.5 Workspace Level
- Workspace-level work makes locating relevant objects and tracing their relationships part of the responsibility, including reconciling evidence and versions and preserving consistency across affected objects.Dependencies can connect customer history to orders, announcements and emails to courses, or repository files and functions to an issue [148] [149] [150] [151].Source: §3.5 · p. 22, §3.5.2 · p. 22
3.6 Synthesis: Work Scope and Capability Demands
- Across the five forms, work complexity scales with the web of dependencies capabilities must carry through decisions, changes, feedback, and revision.At the Workspace Level, agents track sources and dependencies and check consistency across affected objects when updates are required.Source: §3.6 · pp. 22–23
4 Optimization: From Work Experience to System Capability
Work-agent optimization uses experience from work to improve system capability at the model, harness, or coordination level. The process encompasses learning signals, separate model and harness updates, and adaptation of the two together.
4 Optimization: From Work Experience to System Capability
- Work experience supplies observations, action trajectories, and outcome feedback that can inform work-agent improvement.Section 4.1 examines environments, data, and feedback as learning signals.Source: §4 · p. 23
- Model optimization uses pre-training, mid-training, and post-training to develop model capabilities, while harness optimization improves interfaces, workflows, and memory.Model updates improve interpretation, reasoning, and action selection; harness updates enhance tools and execution workflows that support the model.Source: §4 · p. 23
- Joint optimization examines how models and harnesses adapt together and support continued self-improvement.Source: §4 · p. 23
4.1 Training Environments, Data, and Rewards
Work-agent optimization draws on environments, curated corpora and trajectories, and feedback to develop reliable capabilities. The approaches construct work-grounded tasks, preserve action dependencies, and assess outcomes with checks suited to the output and its requirements.
4.1.1 Work Environment Construction
- Training environments combine an initial workspace, available actions, a request, and result checks; construction should preserve dependencies across real work materials.Methods ground tasks in work records, evidence across retrieved files, expert workflow plans, and artifact transitions.Source: §4.1.1 · pp. 25–26
- 33.33% versus 12.83% AutomationBench success with composed versus single-service environments, using the same backbone, training procedure, and 1K SFT trajectories.CompoWorld controls cross-service dependencies while varying the number of services.Source: §4.1.1 · p. 26
- With 1K, 3K, and 5K trajectories, WorkBorne’s broader artifact coverage performs better than more transitions per corpus; at 8K, results are similar.WorkForge reports that increasing training data from 1K to 16K trajectories improves ALE by 13.0 points and WildClaw by 8.4 points.Source: §4.1.1 · p. 26
- 22.11 ALE with combined programmatic and semantic checks, versus 15.62 with semantic-only and 14.89 with programmatic-only selection in WorkForge’s matched 1K-trajectory ablation.GraphForge’s corruption audit finds that deleting a worksheet reduces the target rubric score by 0.377, while finer errors change scores by less than 0.03 on a 0–1 scale.Source: §4.1.1 · p. 26
4.1.2 Corpus and Trajectory Curation
- Curated corpora and trajectories retain work materials, observations, actions, tool results, and failed attempts so models can learn actions and their context.Task, workspace, and trajectory identifiers support coverage analysis and splits by shared sources; interactive tasks also need records of changing requirements and other participants’ actions.Source: §4.1.2 · p. 27
4.1.3 Feedback Design
- Feedback should match the output: direct state checks assess values and feasibility, while content and presentation may require rubrics, preferences, or semantic assessment.Requirements specify success and verifiers assess supporting evidence; intermediate checks and dependency records can connect verified progress to earlier decisions.Source: §4.1.3 · pp. 27–28
- Feedback must track changing requirements and account for verification cost, including which checks need repeating as inputs or artifacts change.Structural and state checks can detect common errors efficiently, while semantic inspection and human assessment address requirements needing more context.Source: §4.1.3 · pp. 27–28
4.2 Model Optimization
Model optimization develops work agents’ knowledge and decision policies through pre-training, mid-training, and post-training. The section describes how each stage supplies different knowledge or feedback, with reported results illustrating the effects of selected training choices.
4.2.1 Pre-training
- Pre-training builds broad semantic and computational knowledge, while work materials can expose structured relationships such as formulas, cell addresses, and computed outputs.The section describes next-token prediction as the standard pre-training objective for autoregressive language models.Source: §4.2.1 · p. 28
- 28.94% SimpleQA accuracy for ten distinct rephrasings used once, versus 23.76% for ten repetitions of the original text, in Kimi K2’s early-checkpoint experiment.The authors suggest extending this approach to aligned views of work content, such as source cells, rendered charts, and report passages.Source: §4.2.1 · p. 29
4.2.2 Mid-training
- Mid-training enriches specialized and long-tail knowledge, introduces tool-interaction patterns, and develops long-context use before later task optimization.Replay is used in described approaches to preserve broader capabilities while increasing exposure to specialized work knowledge.Source: §4.2.2 · p. 30
- With subsequent SFT held fixed, Nemotron mid-training improves SWE-bench Verified Pass@1 from 60.7% to 64.5%.Its mixture includes repositories, pull requests, edit histories, and build logs, with agent traces and pre-training replay.Source: §4.2.2 · p. 30
- Under a fixed 40B-token budget and identical later SFT, TrajLong’s retrieval, information-combination, and file-state reconstruction tasks outperform raw-trajectory training on all seven reported ablation benchmarks.The section also reports that function-aware fill-in-the-middle training improves the mean across SWE-bench Verified and Lite from 13.95 to 15.60 in a controlled 7B comparison.Source: §4.2.2 · p. 30
- In Nemotron’s 100B-token ablation, masking tool responses while retaining them as context raises SWE-bench Verified Pass@1 from 27.5% to 37.8% after identical SFT.The authors note that tool responses provide evidence for later decisions even when the model need not generate them.Source: §4.2.2 · p. 30
4.2.3 Post-training
- Post-training uses demonstrations, execution rewards, teacher guidance, and accumulated feedback to supervise decisions, trajectories, artifacts, or prior attempts.These approaches provide different signals for learning task behavior and acceptable deliverables.Source: §4.2.3 · pp. 30–32
- The Mercor and SkyRL recipe’s 397B model improves held-out APEX-Agents Pass@1 from 16.11% to 27.29%.The recipe emphasizes environment repair and exact token histories; prompt-group weighting reduces the influence of unusually long trajectories.Source: §4.2.3 · p. 31
4.3 Harness Optimization
Harness optimization improves the external support that enables agents to inspect information, act on workspaces, and manage execution across steps. The section covers interfaces, workflows, and reusable experience as distinct targets for improvement.
4.3 Harness Optimization
- Harness optimization targets observation and action interfaces, execution workflows, and reusable experience to address missing evidence, unreliable tools, and poorly organized procedures.Execution traces can reveal where these components need improvement.Source: §4.3 · p. 33
4.3.1 Observation and Action Interfaces
- Observation interfaces should provide evidence needed for the next decision without consuming excessive context or requiring repeated inspection.SWE-agent offers repository operations, while TACO develops and reuses terminal-output filters that preserve relevant state and error information.Source: §4.3.1 · p. 34
4.3.2 Workflows and Execution Control
- Workflow optimization ranges from revising prompts and model-call connections to searching over workflow graphs or modifying harness code.Effectiveness should be measured on subsequent tasks; execution control also needs conditions for continuing, revising results, and deciding when delivery is complete.Source: §4.3.2 · pp. 34–35
4.3.3 Long Context, Memory and Skills
- Context management preserves current requirements, relevant object versions, and unresolved dependencies, while memory and skills let agents reuse strategies and procedures across tasks.Task-specific state and reusable experience have different update schedules and should be kept distinct.Source: §4.3.3 · p. 35
4.4 Model--Harness Coupling and Joint Optimization
Model and harness improvements interact because each shapes the other’s available information, tools, and execution experience. The section reviews harness-aware training, joint adaptation, and self-improvement, and proposes controlled comparisons for assessing their effects.
4.4 Model--Harness Coupling and Joint Optimization
- The paper proposes comparing updated and frozen optimizers on subsequent unseen challenges under fixed acceptance criteria, held-out work, and a shared resource budget.For ongoing work, the proposed evaluation also compares configurations resumed from the same saved workspace after identical changes or interventions.Source: Figure 7 · p. 36, §4.4.2 · p. 38
4.4.3 Self-Improvement and Recursive Self-Improvement
- Cross-task regressions are 0.7% versus 7.0%, and normalized model-token cost is 1.00 versus 0.59, compared with applying all repairs.The passage reports these as a comparison of repair strategies.Source: §4.4 · p. 37
4.4.1 Harness-Aware Policy Training
- Harness-aware training lets policies learn under the observations, tools, and execution rules they will use, so deployment must provide the interfaces and support learned during training.The training harness shapes the situations and decisions a policy learns to handle even when only model weights are updated.Source: §4.4.1 · p. 37
4.4.2 Joint Model--Harness Adaptation
- Joint adaptation alternates or coordinates model updates with changes to prompts, harnesses, memory, or skills, using experience collected with the current version of the other component.A procedure compensating for an earlier model’s limitations may become unnecessary after training.Source: §4.4.2 · pp. 37–38
4.4.3 Self-Improvement and Recursive Self-Improvement
- 70.3% safe repairs for WorkRover, compared with 60.7% for full regeneration, with rates remaining within 70.3–72.4% across three repair generations.WorkRover selects the smallest repair set that satisfies fixed requirements and preserves necessary dependencies, then validates it through fresh execution.Source: §4.4.3 · p. 38
- The VERSE controlled comparison finds the combined setting of execution verification and optimizer self-evolution achieves the best results, though it uses one evolution run per condition.Without executable checks, self-evolution selects the initial harness and produces a lower test result.Source: §4.4.1 · p. 37, §4.4.3 · p. 39
- For self-improvement, the authors propose assessing adaptation while preserving unfinished responsibilities and accepted-work quality; recursive self-improvement additionally compares the updated optimizer with its frozen predecessor.Suggested measures include task success, adaptation costs, verification effort, and recurrence of previously resolved failures.Source: §4.4.3 · p. 39
5 Evaluation and Resources: Assessing Work Agents
Evaluation should determine whether delegated work meets explicit acceptance criteria by examining the relevant deliverables, workspace state, and execution process. The section frames task definitions, evidence, and protocols as necessary context for interpreting benchmark performance.
5 Evaluation and Resources: Assessing Work Agents
- Work-agent evaluation must define delegated work, specify acceptance criteria, and collect evidence appropriate to the deliverable and workspace state.Answers, files, application state, and complete workflows have different assessment requirements.Source: §5 · pp. 39–40
- Benchmark scores should be interpreted alongside the tasks and protocols that produced them because performance depends jointly on the model, tool interface, execution environment, and verification procedure.The section covers task construction, work-result evaluation, metrics, execution infrastructure, and relationships among performance, cost, and economic value.Source: §5 · p. 40
5.1 Benchmarks and Task Construction
Work-agent benchmarks vary both in how tasks are constructed and which domains and work objects they cover, shaping their realism, controllability, and acceptance requirements. Construction approaches trade precise control for fidelity to complete professional delegation.
5.1 Benchmarks and Task Construction
- Benchmarks are scoped by task construction and domain coverage, which shape task realism and controllability, and the work’s representativeness.Construction determines how requests, materials, and execution conditions are produced; domain coverage determines the professional knowledge, objects, and acceptance requirements involved.Source: §5.1 · p. 40
5.1.1 Task Construction Strategies
- Expert-designed tasks offer precise requirements, real-request tasks preserve authentic work objects, interactive environments provide executable context, and workspace tasks approximate complex delegation at higher construction and verification cost.Together, the four approaches trade controllability for realism.Source: §5.1.1 · p. 40
5.1.2 Domain Coverage and Work Requirements
- Acceptance evidence depends on work objects: professional judgments need factual and rule-based correctness, office deliverables need content and structural checks, and enterprise operations need permission and constraint checks.The passage distinguishes acceptance requirements across finance, law, healthcare, office/document work, and customer service or enterprise operations.Source: §5.1.2 · p. 42
- Software, data-science, and research outputs are commonly checked through executable or reproducible results, while CAD and 3D work also requires geometric fidelity and rendering quality.Web, desktop, mobile, and API settings are interaction settings that assess post-operation environment state rather than professional knowledge.Source: §5.1.2 · p. 42
- Cross-domain benchmarks combine work objects and acceptance requirements, spanning localized question answering through complete professional tasks.Figure 8 groups benchmarks by primary work domain or setting and summarizes representative knowledge requirements.Source: §5.1.2 · p. 42
5.2 Evaluation Methodologies
This section organizes work-agent evaluation around the outcomes and properties to inspect, including artifacts, execution, state changes, workflow-level measures, and requirement satisfaction. It covers complementary judgment methods and criteria such as task completion, correctness, delivery completeness, quality, and acceptance.
5.2.1 Evaluation Targets and Evaluation Dimensions
- Evaluation separates the target—the answer, artifact, environment state, or workflow judged at final acceptance—from dimensions such as correctness, completeness, structure, visual quality, and constraint compliance.Answer checks can miss errors in the operations that produced a correct response; artifact checks require structural and functional tests, while state checks must catch unintended changes. Workflow evaluation covers final results, intermediate requirements, and dependencies, with process constraints made explicit.Source: §5.2.1 · pp. 42, 44, §5.2 · p. 42
5.2.2 Evaluation Methods
- Benchmarks judge collected evidence with rules, LLM judges, agent judges, or occupational experts, often combining methods for open-ended work.Rules are clear and repeatable but cover only tested requirements; LLM and agent judgments handle semantic or dispersed evidence but can be subjective or unstable. WildClawBench combines deterministic checks with an LLM or VLM judge, while expert judgment remains a reference for open-ended professional work.Source: §5.2.2 · p. 44
- Scientific evaluations assess discovery correctness and hypothesis matching alongside program executability and computed results.Reported measures include an LLM Hypothesis Matching Score and scientific task completion.Source: §5.2.2 · p. 47
- CAD and 3D-generation evaluations score artifact fidelity, spatial and instruction compliance, and script validity, including CADBench [85] and BlenderBench [130].Metrics include attribute, spatial, instruction, and average scores, syntax errors, IoU, Chamfer Distance, and valid shape rate.Source: §5.2.2 · p. 47
- Digital-interaction and workflow benchmarks evaluate tool-use correctness, task completion, milestone progress, coordination, and compliance with target-state constraints.Measures include action-type and multi-turn accuracy, task and milestone scores, and long-horizon completion with cross-step constraints.Source: §5.2.2 · p. 48
- Professional-work evaluations assess requirement satisfaction, deliverable completeness, professional quality, expert acceptance, and economic value.Metrics include LLM Expert Score, Pass Rate, Economic Value, Pass@1 and Pass@8, and expert Win Rate or Win-or-Tie Rate.Source: §5.2.2 · pp. 48–49
- Evaluation considers end-to-end remote work products across papers, code, data, and websites.The passages name these artifact types but do not specify additional evaluation criteria for them.Source: §5.2.2 · p. 49
- Expert assessment uses automation rate and checks requirement satisfaction, task completion, and dependency-relation correctness.Source: §5.2.2 · p. 49
- Workflow evaluation covers multi-file workspaces and uses rubric pass rate, task completion rate, Node F1, and Edge F1.Source: §5.2.2 · p. 49
- For professional workspaces and complete delivery tasks, evaluation emphasizes importance-tiered requirement satisfaction and delivery completeness.Source: §5.2.2 · p. 49
5.3 Evaluation Metrics
Work-agent metrics aggregate task judgments through outcome scores, relative comparisons, or explicit weighting. Their scores require context about units, thresholds, baselines, and weights before they can be interpreted across benchmarks.
Task outcome scores
- Task outcome scores measure acceptance or completion, but their differing units and strictness make scores not directly comparable across benchmarks.Examples include task success, answer accuracy or matching, rubric scores, continuous outcomes, pass@k, and passk.Source: §5.3 · p. 49
Relative comparison metrics
- Relative comparison metrics use preferences, win rates, or rankings when no unique reference answer exists, and their interpretation depends on the comparison set and method.GDPval uses blind occupational-expert comparisons; CentaurBench aggregates pairwise LLM-judge judgments into separate automation and augmentation rankings.Source: §5.3 · p. 49
Weighted performance metrics
- Weighted performance metrics aggregate outcomes using explicit weights on criteria or tasks, but remain performance scores rather than measures of economic return.PRBench weights expert criteria; OmegaUse-OfficeVal weights tasks by recorded human labor time or a task-price proxy.Source: §5.3 · pp. 49–50
Metric interpretation
- Metrics should align with accepted work objects and quality requirements and state thresholds, partial credit, baselines, and weight sources rather than imply equivalent capability scales.Table 11 summarizes the computation and interpretive scope of the metric classes.Source: §5.3 · p. 50, Table 11 · p. 50
5.4 Open-Source Harnesses and Execution Environments
Work-agent evaluation depends on both the agent harness and the evaluation harness, as well as on execution environments suited to the work objects being tested. Table 12 summarizes representative benchmark environments and verification mechanisms.
- Agent harnesses manage context, tool invocation, and execution control, while evaluation harnesses initialize tasks, record runs, and verify results.Source: §5.4 · p. 50
- Software-engineering tasks use code repositories and test environments, web and desktop tasks use resettable interactive environments, and enterprise tasks check business state through APIs and databases.Office and design tasks require verification of native files, computed results, and deliverables.Source: §5.4 · pp. 50–51
- Table 12 summarizes the execution environments and verification mechanisms of representative work-agent benchmarks.Source: Table 12 · p. 51, §5.4 · p. 51
- Benchmark scores should be interpreted alongside the execution framework, tool configuration, and resource budget, rather than attributed entirely to the underlying model.HAL and Harness-Bench examine model–harness effects on performance and execution behavior [324] [325].Source: §5.4 · p. 52
5.5 Performance, Cost, and Economic Value
Evaluating work agents requires separating execution cost, the potential value of professional work, and deployment return after human oversight. The section surveys measures for these dimensions and argues that performance alone does not capture practical utility.
5.5.1 Trading Performance against Execution Cost
- Task performance and execution cost can diverge, and harness choices can change invocation cost even when success rates are close.HAL compares scores and invocation cost using a Pareto frontier, while HarnessTax reports cost differences across harnesses for the same model [324].Source: §5.5.1 · p. 53
5.5.2 The Value of Professional Work
- Work-value evaluation must account for task representativeness and professional acceptance, rather than equating annotation pay or human working hours with actual economic return.Benchmarks connect tasks to market pay, expert comparisons, paid projects, artifact quality, completion time, and task-price proxies [279] [288].Source: §5.5.2 · p. 53
5.5.3 Reliable Deployment and Agentic ROI
- Deployment utility depends on delivery quality, human time saved, and use costs, including users’ effort to describe, check, and revise work.READY finds that agents with comparable autonomous accuracy may require different amounts of human assessment to meet the same deployment requirement [330].Source: §5.5.3 · p. 53
- Economic evaluation should distinguish task execution resources, the potential value of produced work, and deployment return after human oversight.Performance–cost frontiers, harness tax, and agentic ROI complement task-performance evaluation from the perspectives of efficiency, execution configuration, and user utility.Source: §5.5.3 · p. 53
6 Frontiers: Toward Next-Generation Work Agents
The research agenda for next-generation work agents addresses continuing user goals, representation of individual judgment, and coordination across organizational roles, with technical advances and evaluation to sustain reliable work as conditions change. It also emphasizes targeting whichever execution, assessment, integration, or human-decision stage limits delivery in pursuit of sustainable progress in useful professional work.
6.1 Future Applications: From Assistance to Delegated Work
- Personal agents sustain user goals, digital twins represent a particular professional’s judgment, and organizational systems coordinate interdependent roles toward shared delivery.These applications require agents to retain open goals and resume work, distinguish professional competence from fidelity to a person, and manage dependencies, decision rights, handoffs, and conflicts.Source: §6 · pp. 53–54, §6.1.1 · p. 55, §6.1.2 · p. 55
6.2.1 Environment: Learning across Realistic Work Worlds
- Work environments should represent realistic occupational and workspace variation, support reproducible interrupted episodes, and allocate practice toward difficult professional capabilities.Decision-relevant dependencies, histories, competing versions, shared state, and incomplete updates can exercise clarification and repair; learning gains and transfer can guide costly practice choices.Source: §6.2.1 · pp. 56–57
6.2.2 Persistence: Memory, Initiative, and Adaptation
- Persistent delegation requires memory grounded in current work, initiative guided by retained responsibilities and authority, and learning from repeated cycles.Memory must track who, role, source, and applicability period; feedback should distinguish repairs to current work, reusable procedural improvements, and changes to a represented person’s judgment.Source: §6.2.2 · pp. 57–58
6.2.3 Verification: Work Quality, Personal Fidelity, and Return on Delegation
- Verification should separately assess whether deliverables meet their purpose and whether a digital twin reflects the represented person’s decisions and agreed authority.Acceptance combines direct checks with expert judgments about purpose and audience; fidelity uses held-out decisions under comparable information and remains distinct from work quality.Source: §6.2.3 · p. 58
- Beneficial delegation requires tracing total human and computational effort and whether accepted work proves useful in its downstream context.Effort includes specification, supervision, correction, and integration; comparisons with alternative ways of completing the work help attribute benefits to delegation, while version-linked evidence identifies results needing rechecks.Source: §6.2.3 · pp. 58–59
6.3 Two Interacting Loops: Delivering Work and Improving Capability
- Execution and capability improvement form interacting loops: persistent work records support ongoing responsibilities, while execution history and downstream feedback guide tested updates.Updates should be assessed on independent work settings, and their effects on current results and commitments should inform whether to apply them, migrate affected work, or retain the current procedure.Source: §6.3 · p. 59, Figure 10 · p. 59, §6.3.1 · p. 59
6.3.3 Coupling Learning to Open Responsibilities
- Coupling learning to open responsibilities means tracing an update through dependencies to determine which results to recompute, checks to repeat, and handoffs to reopen.Studies should vary update scope and timing during unfinished work and compare gains in accepted, used work against training, inference, rechecking, repair, and coordination costs.Source: §6.3.3 · p. 60
6.4 Research Outlook: Useful Work under Openness and Persistence
- Research should test openness and persistence separately and together to establish where agents reduce existing work effort or make previously inaccessible work feasible.Across work conditions, assessment should combine accepted work, value in use, and total effort over the delegation period; persistent service can reuse context but requires maintenance, while open work adds discovery and clarification.Source: §6.4 · pp. 60–61, Figure 11 · p. 61
- As execution improves, assessment, integration, or human decisions may become limiting stages; the two loops should target those constraints and enable sustainable, accepted, useful professional work.The execution loop organizes work around the limiting stage; the improvement loop addresses the capability or coordination constraining the next delivery.Source: §6.4 · p. 61
7 Conclusion
The conclusion frames reliable digital work as maintaining connections among requirements, actions, and outcomes as work unfolds. It links execution, capability improvement, and evaluation to sustained delegation under changing conditions.
- As work expands from answers or local changes to workflows and related workspace objects, agents must track dependencies and use each step’s results to guide the next.Openness makes discovery part of this process, while persistence carries responsibility across stages and events.Source: §7 · p. 61
- Work execution supplies observations, procedures, corrections, and feedback that can improve subsequent work, while training and harness changes can develop reusable patterns and improve evidence access, control, and retained state.Source: §7 · p. 61
- Evaluation checks whether answers, artifacts, state changes, and workflow results meet requirements, while separating the inspected result, assessed properties, judging method, and reported metric clarifies what an improvement advances.Source: §7 · p. 62
- Sustained delegation requires agents to adapt and improve while completing work under existing agreements, with gains weighed against dependent-work repair and update effort.The paper identifies continuing goals, fidelity to an individual’s judgment, and coordination toward joint delivery as distinct settings for testing this capability.Source: §7 · p. 62
Related papers
This paper cites
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?2023
- Let's Verify Step by Step2023
- Self-Refine: Iterative Refinement with Self-Feedback2023
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018
- Generative Agents: Interactive Simulacra of Human Behavior2023
- Voyager: An Open-Ended Embodied Agent with Large Language Models2023