PaperlayerView exact revision on arXiv

Source-linked AI summary

JobBench: Aligning Agent Work With Human Will

Yuetai Li, Yichen Feng, Zhangchen Xu, Zixian Ma, Kaiyuan Zheng, Fengqing Jiang, Xinghua Sun, Rulin Shao, Zichen Chen, Yue Huang, Xinyang Han, Brian Lee, Kayla Xu, Shenglai Zeng, Hang Hua, Xiangliang Zhang, Basel Alomair, Ranjay Krishna, Luke Zettlemoyer, Pang Wei Koh, Bhaskar Ramasubramanian, Luyao Niu, Xiang Yue, Radha Poovendran

arXiv:2605.26329v1cs.AI

TL;DR

Existing workplace benchmarks emphasize economically valuable deliverables, while giving less attention to which duties workers prefer to delegate. JobBench evaluates tasks derived from those preferences using heterogeneous reference files and chained rubrics; across 36 configurations, the strongest scores 45.9%.

  • Problem

    Workplace benchmarks have focused on economically valuable deliverables, while worker preferences about which duties to delegate are a complementary criterion for evaluating workplace agents.

    Source: §1 · p. 1

  • Method

    JobBench builds 130 tasks across 35 occupations from Workbank delegation preferences, with heterogeneous reference files graded by chained, fact-anchored rubrics.

    Source: §1 · p. 2

  • Results

    45.9% is the score of Claude Opus 4.7 under Claude Code, the strongest of 36 evaluated configurations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    JobBench tracks agent capability on duties workers most want delegated, aligning the evaluation target with worker satisfaction and productivity.

    Source: §1 · p. 2

    The benchmark covers U.S.-centered, digital, document-heavy tasks in 35 selected occupations and is designed for evaluation, not deployment validation.

    Source: Limitations · p. 15

Abstract

from arXiv · show

Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.

1 Introduction

Workplace benchmarks have largely measured economically valuable work agents can deliver, while leaving workers’ delegation preferences out of the evaluation. JobBench instead builds tasks from duties experts want delegated, then tests professional reasoning with source files and chained rubrics.

1 Introduction

  • Unlike benchmarks centered on economic value, JobBench treats workers’ stated delegation preferences as a human-centered constraint on which duties merit automation.The framing positions professionals as domain experts whose preferences inform work intended to enhance productivity.Source: §1 · p. 1
  • JobBench builds 130 tasks across 35 occupations from Workbank survey responses on workers’ preferences for delegating their work duties.The survey covered more than 1,500 workers; the selected occupations had high average willingness for automation.Source: §1 · p. 2
  • JobBench grades tasks with chained rubrics averaging 35.6 binary criteria, awarding credit only when every criterion in a chain passes.The criteria are anchored to deterministic numbers, reasoning steps, or documented professional judgments; a correct fact reached through a wrong inference earns no partial credit.Source: §1 · p. 2
  • Each task presents heterogeneous reference files that may conflict or hint at where to search, rewarding agents that retrieve and reconcile sources.The example duty is checking different source materials to obtain relevant facts.Source: §1 · p. 2
  • 45.9% is the score of Claude Opus 4.7 under Claude Code, the strongest of 36 evaluated agent configurations.The introduction also reports that no agent outside the Claude and GPT families exceeds 19%.Source: §1 · p. 2

2 The JobBench Benchmark

JobBench centers expert preferences for delegation and evaluates source-grounded workplace reasoning across tasks developed, filtered, and graded through expert-informed processes.

2.1 Design Principles of JobBench

  • JobBench prioritizes duties experts want delegated, tests source-grounded reasoning in heterogeneous workspaces, and frames automation as augmentation rather than substitution.For reporters, the WorkBank [34] preference is cross-source fact checking; JobBench requires cross-year reconciliation, unlike GDPVal’s single-article edit [31].Source: §2.1 · p. 3

2.2 JobBench Overview

  • The benchmark contains 130 tasks across 35 occupations and 10 SOC groups, backed by 502 reference files in 17 formats.The main set has 65 tasks; the easy set has 65, and all easy-set reference files come from the real world.Source: §2.2 · p. 3, Figure 2 · p. 4
  • Each task combines a professional query, heterogeneous reference files, binary checks, and reasoning-chain rubrics awarded only when every node passes.Rubric criteria are anchored to verifiable facts, numbers, or documented professional judgments.Source: §2.2 · p. 4

2.3 Benchmark Construction

  • Occupation selection combines Workbank delegation-desire scores with wage-based economic exposure, then filters duties for digitalizability, evaluability, and supportability.The 35 selected occupations have average desire scores above 3 and are ranked by economic exposure.Source: §2.3 · p. 4
  • Experts recruited through Prolific and Upwork inform task sketches, which are expanded with AI assistance into workspaces, queries, and rubric chains.Easy tasks require no web-search evidence and pose fewer reasoning challenges than Main tasks.Source: §2.3 · pp. 4–5, Figure 3 · p. 5
  • Rubric criteria must be self-contained, binary, objective, and unambiguous; the authors identify ambiguity as key to LLM-judge disagreement.Unambiguous criteria name the referent precisely enough that two graders cannot reasonably disagree.Source: §2.3 · p. 5
  • Candidate tasks pass automated audits, annotator review, and solve trials; only tasks whose union of passed rubrics exceeds 90% are retained.Audits check instruction-file consistency, professional plausibility, and rubric correctness.Source: §2.3 · p. 5
  • 95.4% of criteria were passed by at least one agent on one sample across the accepted benchmark, after 71% of tasks passed the three-stage quality check.The authors present the union pass rate as evidence that the rubric set is achievable in practice.Source: §2.3 · p. 5

3 Experiments

The experiments evaluate JobBench across model–scaffold configurations and compare its difficulty with GDPVal and an easier task set. They also examine how research and startup attention aligns with occupational capability and worker automation desire.

3.1 Experimental Setup

  • The evaluation spans recent Claude, GPT, Gemini, Qwen, MiniMax, Kimi, and Grok models across four agentic scaffolds.The scaffolds are Claude Code, Codex CLI, OpenCode, and OpenClaw; each uses its maximum supported default reasoning effort.Source: §3.1 · pp. 5–6

3.2 Main Results

  • On JobBench, the strongest configuration, Claude Opus 4.7 under Claude Code, reaches 45.9; next are GPT-5.5 under Codex at 42.7 and GPT-5.4 under Codex at 38.9.Source: §3.2 · p. 7
  • GPT-5.4 under Codex scores 83.0 on GDPVal but 38.9 on JobBench-Main, where runtime is 2.40× GDPVal and tool calls and trajectory events are roughly 1.3×.GDPVal scores are win+tie rates against industry-expert reference deliverables.Source: §3.2 · p. 7, Figure 4 · p. 7
  • JobBench-Main scores are 26–31 points below the easy set, and GPT-5.4 under Codex takes 16.4 rather than 6.4 minutes.The easy set has fewer reasoning challenges, multi-source conflicts, and deliverable files, and requires no web search.Source: §3.2 · p. 7
  • GPT-5.4 under Codex gains +7.0 JobBench-Main points as reasoning effort rises from low to xhigh.The performance gain is monotonic across the tested reasoning-effort levels.Source: §3.2 · p. 7, Figure 6 · p. 8
  • Scaffold choice shifts scores for a fixed model; GPT-5.4 scores 38.9 under Codex versus 34.6 under OpenCode, with the largest gaps in three occupations.The largest induced gaps are in Office / Admin Support, Computer / Mathematical, and Management.Source: §3.2 · pp. 7–8
  • Grok-4.1-Fast judge scores agree with Claude Opus-4.5 within 0.7 point variance while reducing full-run judging cost from $38–$46 to $1.97.The reported reduction is approximately 20×.Source: §3.2 · p. 8, Figure 5 · p. 8

3.3 Occupational Analysis

  • Research papers and startups both concentrate more on the low-capability R&D quadrant than the high-capability Sweet Zone, with attention ratios of 1.56 and 1.62.Research leans toward knowledge-heavy occupations, while startups concentrate more on customer-service representatives and financial managers.Source: §3.3 · pp. 8–9, Figure 7 · p. 9

4 Conclusion

JobBench evaluates agent capability on work duties workers want delegated, and the strongest of 36 evaluated models reaches 45.9. The authors position it as a way to track work aligned with worker satisfaction and productivity.

4 Conclusion

  • Claude Opus 4.7 under Claude Code reaches 45.9, while no models outside the Claude and GPT families exceed 20%.The paper presents JobBench as a leaderboard for tracking capability on duties whose automation would most directly enhance workers’ satisfaction and productivity.Source: §4 · p. 9
  • JobBench covers 130 tasks across 35 occupations, each based on a Workbank-elicited delegation preference and packaged with heterogeneous reference files.Its chained rubrics contain 4,631 binary criteria and award credit only when every step in the chain holds together.Source: §4 · p. 9
  • The authors hope JobBench shifts the labour-market target from replacement toward enhancement by building agents for work people want delegated.They frame the benchmark as tracking agent capability on duties whose automation would most directly enhance workers’ satisfaction and productivity.Source: §4 · p. 9

B Limitations, Ethics, Broader Impact, and LLM Usage . . . . . . . 15

The listed supplementary sections include a leaderboard and a task split by occupation.

B Limitations, Ethics, Broader Impact, and LLM Usage

E Representative Task Examples

The supplementary contents list representative task examples from four occupations and templates for agent-evaluation and judge prompts.

E Representative Task Examples

  • Representative task examples are listed for Reporters and Correspondents, Data Entry Keyers, Lawyers, and Web Administrators.The contents place these examples in Appendix E, with entries beginning on pages 18, 22, 26, and 31, respectively.Source: E Representative Task Examples · p. 13

F Prompt Templates and Evaluation Interface

G Supplementary Analysis on JobBench’s Position in AI Labour Market

The discussion situates JobBench among knowledge, interactive-agent, workplace, and labour-market benchmarks. It distinguishes JobBench by evaluating workflows experts identify as high-priority for delegation and by building benchmark task packages from worker preference signals.

A Related Work

  • Other agentic benchmarks cover issue resolution, enterprise web stacks, desktop environments, and assistant or tool-use settings, while exam-style benchmarks assess knowledge and reasoning.Examples include SWE-bench, WebArena and WorkArena [45] [12], OSWorld [40], MMLU [15], GPQA, and MMLU-Pro [38].Source: A Related Work · p. 14
  • Unlike workplace benchmarks scoped primarily by economic value, JobBench evaluates workflows experts identify as high-priority for delegation.The passage contrasts this with GDPVal [31], PRBench [1], and the Remote Labor Index [18].Source: A Related Work · p. 14
  • Workplace benchmarks include GDPVal [31] for economically valuable tasks, PRBench [1] for legal and financial reasoning, and the Remote Labor Index [18] for remote-work projects.Source: A Related Work · p. 14
  • Workbank [34] surveys over 1,500 U.S. workers on O*NET duties for delegation preferences and desired human-AI collaboration modes; JobBench turns these signals into task packages.Source: A Related Work · p. 14

Limitations

JobBench is limited to selected U.S.-centered, digital, document-heavy professional work and is intended as a benchmark, not deployment validation. Its broader-impact framing favors augmentation while warning against interpreting results as evidence of worker-replacement readiness or safe high-stakes deployment.

Limitations

  • The benchmark covers 35 selected O*NET occupations and excludes physical work, non-U.S. labor markets, non-English workplaces, real-time collaboration, and long-term organizational workflows.It is not deployment validation and is not recommended for high-stakes professional judgments or operation without human oversight.Source: Limitations · p. 15

Ethics Statement

  • Human-participant annotation was voluntary, exempt-reviewed, and collected no sensitive personal information; only aggregated results and no identifiable participant data are released.Participants consented, and the paper commits to fair compensation under occupational wage standards.Source: Ethics Statement · p. 15

Broader Impact

  • The study frames its intended impact as evaluating work professionals want help with, while warning that the dataset could be misused to claim replacement readiness or safe deployment.Its U.S.-centric, document-heavy scope may underrepresent non-U.S., non-English, physical-labor, and low-resource contexts.Source: Broader Impact · p. 15

C Leaderboard

  • Figure 8 ranks 15 models on the JobBench main set, with model-specific coding scaffolds used for the listed model families.The supplied passages give no leaderboard scores or ranking outcomes.Source: C Leaderboard · p. 16, Figure 8 · p. 16

D Task Split by Occupation

The appendix reports how tasks are split by occupation across the JobBench main and easy sets. Table 4 presents the per-occupation counts.

D Task Split by Occupation

E Representative Task Examples

The appendix presents four representative JobBench tasks from the main set, using three views for each task. These examples cover reporting, data entry, law, and web administration.

E Representative Task Examples

  • Four main-set examples cover Reporters, Data Entry Keyers, Lawyers, and Web Administrators.Each example includes a one-page schematic, verbatim agent instruction, and reviewer-facing rubric and task-card parse.Source: E Representative Task Examples · p. 18

Reporters and Correspondents – Task 1 Pre-publication editorial package

The reporter task asks an agent to prepare an editorial package on lead in drinking water and childhood health using heterogeneous source materials. It requires cross-source analysis, source verification, and careful handling of conflicting evidence and data limitations.

Reporters and Correspondents – Task 1 Pre-publication editorial package

  • The workbook must analyze 2020–2024 water-lead trends, compare Connecticut blood-lead data with national totals, and map five water-system cities to town-level surveillance data.It must calculate trends and flag threshold exceedances; the CDC dataset has Connecticut submissions only for 2017–2019.Source: Reporters and Correspondents – Task 1… · pp. 18–21
  • The pitch memo must connect each system’s lead-service-line counts to the LCRI’s 10-year replacement mandate and compare Waterbury’s trajectory with Newark’s crisis.The task calls for a structural-trajectory parallel, including sustained exceedances, delayed enforcement, and vulnerable populations, rather than a point-in-time comparison.Source: Reporters and Correspondents – Task 1… · pp. 20–21
  • The task asks a Connecticut investigative reporter to produce a thesis-driven pitch memo, a three-sheet analysis workbook, and a source log with at least 15 entries.The package draws on FOIA water data, state surveillance, federal data, regulatory materials, factsheets, and interview notes.Source: Reporters and Correspondents – Task 1… · pp. 18, 20, 21
  • The task requires reconciling the surveillance report’s finding that no investigated homes identified water as a lead hazard with water-system readings above the 15 ppb action level.The reasoning must account for the difference between home-investigation methods and system-level monitoring.Source: Reporters and Correspondents – Task 1… · p. 21
  • The agent must assess Dr. Martinez’s 30% referral-increase claim against state figures, treating it as directionally consistent but quantitatively conservative and potentially clinic-specific.The task also requires distinguishing the finalized LCRI from when its new 10 ppb action level applies to monitoring.Source: Reporters and Correspondents – Task 1… · p. 21
  • All five cities in the water data appear among the state report’s top 10 high-risk communities, while five other listed communities are absent from the data and must be flagged as a coverage gap.The task also calls for accurately reporting the report’s counterintuitive incidence ordering across racial and ethnic groups.Source: Reporters and Correspondents – Task 1… · p. 21

E.2 Data Entry Keyers: Q3 Sales Order Reconciliation

The task asks a data-entry keyer to reconcile Q3 2025 orders across scanned forms, CRM, and ERP records, then produce audit deliverables. It tests source-priority reasoning, calculation checks, customer-master validation, and classification of recurring error patterns.

Reasoning challenges

  • The error log must record each discrepancy’s source, type, reconciled value, and whether it reflects a systematic pattern or a random error.The task calls for aggregating individual errors into categories such as transpositions, calculation errors, Sales Channel errors, and typos.Source: §E.2 · pp. 22, 25

Audit outputs

  • The findings report must calculate data quality from clean versus discrepant records, while the checklist must fail the revenue-accuracy criterion for documented calculation errors.The rubric specifies 30 total records and discrepancies in 17 orders; it identifies two revenue-calculation errors exceeding the ±$0.01 tolerance.Source: §E.2 · p. 24

Task scope

  • The keyer must reconcile 30 Q3 2025 orders across CRM, scanned forms, and ERP, then deliver a discrepancy log, reconciled master, findings report, and compliance checklist.Scans outrank CRM, which outranks ERP; for orders 0011–0030 without scanned originals, CRM takes priority over ERP.Source: Figure 10 · p. 22, §E.2 · p. 25

Reasoning challenges

  • Three-way reconciliation must distinguish erroneous component fields from totals by applying Total = Units x Price in both directions.Some records have a correct total but an incorrect unit price or unit count, requiring reverse-validation rather than flagging the total.Source: §E.2 · pp. 22, 25
  • The audit must cross-check all 30 orders against the 35-customer master and flag unregistered entities and nonstandard country names.Two orders name entities absent from the customer master; the “Italia”/“Italy” discrepancy requires checking the registered country list.Source: §E.2 · pp. 22, 25

E.3 Lawyers: Short-Term Rental Ordinance Settlement Analysis

The lawyer task evaluates a settlement offer affecting six short-term-rental properties through legal analysis and property-specific financials. It calls for a settlement memo, a three-jurisdiction regulatory comparison, and a counter-proposal.

E.3 Lawyers: Short-Term Rental Ordinance Settlement Analysis

  • Three properties face $500-per-day fines, reaching $96,000 in aggregate by January 31, 2026 if settlement is not reached by the waiver deadline.The rubric identifies Properties 1, 3, and 5 as accruing fines and specifies $6,000 per property as of December 10, 2025.Source: §E.3 · p. 27
  • The regulatory table must compare seven provisions across Millbrook, Belvidere Township, and New Orleans, including owner occupancy, night limits, fines, and zoning.The comparison also covers permit caps, spacing, and amortization or grandfathering.Source: §E.3 · p. 27
  • The counter-proposal must seek a longer transition using property-specific financial justification and protect claims through tolling or preserved litigation rights.The rubric points to substantial mortgages and the STR-to-LTR income gap as grounds for an extension, and flags the risk of dismissal with prejudice during negotiations.Source: §E.3 · pp. 27–28
  • The analysis must apply Penn Central’s economic-impact, investment-backed-expectations, and government-action factors to the clients’ circumstances.The rubric describes income reductions, prior STR operations and permits, and the ordinance’s 180-day amortization period and $500 daily fines as relevant facts.Source: §E.3 · p. 28
  • The analysis must assess whether Millbrook’s owner-occupancy rule is vulnerable under the dormant Commerce Clause by comparing it with Hignell-Stark.The rubric notes that the Fourth Circuit has not addressed the issue and treats the Fifth Circuit reasoning as persuasive authority.Source: §E.3 · p. 28
  • The memo must assess whether Millbrook’s 180-day amortization without grandfathering is vulnerable given the cited cases’ different uses and longer periods.The rubric connects amortization adequacy to the clients’ investment-backed expectations and financial circumstances.Source: §E.3 · pp. 28–29

Web Administrators – Task 1 Incident reconstruction and hardening

The web-administrator task centers on investigating an incident, checking an unverified change request against observed attacks and live configurations, and producing hardened configurations and an incident report. The appendix also specifies the evaluation prompts and discusses JobBench’s worker-oriented labor-market framing.

Web Administrators – Task 1 Incident reconstruction and hardening

  • The incident task combines attack-chain reconstruction, change-request compliance auditing, zone-aware hardening, TLS assessment, and a NIST-structured report.The task links observed attack outcomes to configuration gaps and asks for hardened web-server and firewall configurations.Source: Web Administrators – Task 1 Incident… · pp. 31, 35, Figure 12 · p. 31

F Prompt Templates and Evaluation Interface

  • The shared CLI-runner prompt places each task in an isolated temporary workspace and directs agents to its instructions, reference files, and output directory.The prompt requires final deliverables only in the designated output directory.Source: §F.1 · p. 36, F Prompt Templates and Evaluation… · p. 36

F.1 Agent Evaluation Prompt

  • The agent prompt requires following task instructions, using appropriate tools for files that cannot be read directly, and explaining how it resolves conflicting information.It also restricts access to the task workspace or online searches for needed references.Source: Agent Evaluation Prompt · p. 36

Agent Evaluation Prompt

  • The agent prompt specifies workspace boundaries and prohibits saving intermediate or temporary files alongside the final deliverables.It directs agents to save only final deliverables in the designated output directory.Source: Agent Evaluation Prompt · p. 36

F.2 Judge Prompt

  • JobBench evaluates one rubric at a time by supplying its criteria and extracted output contents and requiring a structured JSON judgment.When visual evidence is required, the judge call also attaches output images.Source: §F.2 · p. 36, Judge Prompt · p. 37

Judge Prompt

  • The judge assesses every criterion independently as pass or fail, and a rubric passes only when all its criteria pass.Judgments must include specific output-file evidence in the required JSON structure.Source: Judge Prompt · p. 37

G Supplementary Analyses on JobBench’s Position in the AI Labour Market

  • JobBench measures delegation demand at the task level, where worker preferences and current AI capability may diverge.The supplementary discussion says current AI can speed routine work but struggles with judgment-heavy details needed for reliable delegation.Source: G Supplementary Analyses on JobBench’s… · p. 38
Loading 2605.26329v1…