Source-linked AI summary
DAYJOB: A Benchmark for Long-Horizon Professional Work
Stephanie Finley, Liudas Panavas, Thomas Mikkelson, Cam Hinton, Stacey Ganss, Bradley Monton, Emily Kendall, Michelle Spradlin, Lydia Bye, Michael O'Brien, Lauren Ylvisaker, Derek Ray, Suhaas Garre, Sushant Mehta, Edwin Chen
TL;DR
Professional requests can leave agents to infer what work is needed and whether the premise is sound, even when files contradict it. DAYJOB evaluates this work with realistic workspaces and expert rubrics; strict pass rates are low across evaluated configurations, and case studies show wrong premises carried into analyses.
Problem
Professional requests often leave unclear what work is needed, which documents matter, or whether the request’s premise holds.
Source: §1 · p. 1Method
Results
24.7% of healthcare and 23.9% of finance attempts pass for Claude Opus 5.5, while median configurations pass 0.6% and 2.5%, respectively.
Source: §7 · p. 8Takeaways & Limitations
Failed attempts identify the fact or decision missed, and case studies show agents accepting contradicted premises or carrying wrong inputs through consistent analyses.
Source: §7 · p. 8Because criteria are unweighted, a 95% pass can miss a decisive criterion, such as a required release hold.
Source: §4.4 · p. 5
Abstract
from arXiv · showhide
Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the request's premise holds. We introduce DAYJOB, a benchmark of 130 tasks built by professionals in healthcare (50) and finance (80). The tasks are estimated to take a professional 13.6 hours on average in healthcare and 16.6 in finance. Each task is a containerized Harbor environment with an expert rubric of binary criteria (median 47.5 and 57.5 per task) that an agentic judge applies to the delivered files, and an attempt passes only if it meets every criterion. Across 30 model configurations from 13 developers, the strongest, Claude Opus 5.5, passes 24.7% of healthcare and 23.9% of finance attempts, and the median configuration passes 0.6% and 2.5%. In case studies, agents accept premises that the record contradicts and carry wrong inputs through otherwise consistent analyses. We release all healthcare tasks, 50 of the 80 finance tasks, the evaluation harness, and the leaderboard.
1 Introduction
Professional requests can leave crucial context unstated, and their premises may conflict with the record. DAYJOB pairs such requests with realistic workspaces and expert rubrics; model configurations rarely pass every criterion.
1 Introduction
- A fund’s price entered in rand instead of South African cents inflates its apparent value, while a patient’s chart conflicts with the request’s Bell’s Palsy premise.The healthcare rubric requires ruling out arterial dissection before discharge.Source: §1 · p. 1
- GDPval gives models detailed briefs, while existing agent benchmarks focus mostly on software, web, and desktop work, and healthcare and finance benchmarks often test bounded tasks.GDPval also reports worse GPT-5 performance when prompts omit where to find data and how to approach the problem [22].Source: §1 · p. 1
- DAYJOB pairs colleague-style requests with their implicit files and binary criteria for competent deliverables, including actions the request leaves unstated.Tasks run in containerized Harbor environments; an agentic judge checks delivered files and every criterion must be met to pass.Source: §1 · p. 1
- 24.7% of healthcare and 23.9% of finance attempts pass for Claude Opus 5.5 (adaptive/max), while median configurations pass 0.6% and 2.5%, respectively.The release includes all 50 healthcare tasks, 50 of 80 finance tasks, the harness, and the leaderboard.Source: §1 · p. 1
2 Related work
Prior work evaluates professional tasks, interactive agents, and domain-specific healthcare and finance capabilities through different task formats. DAYJOB instead pairs brief requests with large workspaces and requires rubric-complete work products.
Evaluations of economically valuable work
- Economically valuable work benchmarks include GDPval’s professional tasks [22], APEX’s expert-rubric cases [28], and real freelance projects in the Remote Labor Index [15] and SWE-Lancer.TheAgentCompany simulates a software company [32]; DAYJOB gives agents brief requests, large workspaces, and task-specific rubrics that must be met in full.Source: §2 · p. 2
Agent benchmarks
Healthcare and finance
- Healthcare and finance benchmarks cover exams, conversations, simulated encounters, EHR tasks, financial reports, filings, and tool-assisted research.DAYJOB instead ends heterogeneous-record tasks in work products such as chart notes, appeal letters, credit memos, and audit workpapers.Source: §2 · p. 2
3 The DAYJOB benchmark
DAYJOB defines tasks as requests, file-filled environments, and binary rubrics, designed to test whether agents can infer the work, find relevant evidence, and deliver complete outputs. Its tasks span long professional workflows and undergo review before release.
3.1 Task definition
- A task combines a request, a container environment holding input files, and a rubric of binary criteria; a judge scores each criterion against the deliverables and trajectory.The attempt reward is the fraction of criteria met, with strict passing requiring a perfect score.Source: §3.1 · p. 2
3.2 Design principles
- Requests may leave deliverables implicit, while workspaces mix relevant files with irrelevant, stale, or conflicting sources and rubrics check substance, consistency, and prohibited errors.The shared preamble tells agents to infer their role and deliverables from files without clarification; prohibitions appear in all healthcare and 43 of 50 public finance rubrics.Source: §3.2 · pp. 2–3
3.3 An example task
- A fund-reporting task asks whether reports should be stopped, requiring the agent to reconcile 13 files, catch a cents-versus-rand pricing error, and identify pricing-control issues.Its 20 criteria require quantifying errors, withholding reports until correction, recognizing manual keying as a control weakness, and avoiding false alarms.Source: §3.3 · p. 3
3.4 Composition
- DAYJOB workspaces contain 17 times as many files in healthcare and 22 times as many in finance as GDPval’s gold subset, while most DAYJOB tasks have at least ten files.Rubric sizes are comparable, but every DAYJOB criterion must be met; task creators estimate a median duration of 5–10 hours, with 32% estimated at 20 hours or more.Source: §3.4 · pp. 3–4
- Tasks cover clinical, pharmacy, payer, compliance, valuation, credit, fund-operations, treasury, and audit work.Healthcare and finance ranges of work include deliverables such as medication reconciliation and covenant-compliance reviews.Source: §3.4 · p. 4
3.5 Construction, selection, and release
- Professionals created tasks and each task underwent three review layers; all healthcare tasks and 50 of 80 finance tasks are public, alongside the harness and analysis scripts.The remaining finance tasks are available on request, and dataset cards ask that tasks and derivatives be excluded from training and checkpoint selection.Source: §3.5 · pp. 4–5
4 Evaluation protocol
The evaluation runs agents in constrained task containers, grades their files criterion by criterion, and reports task-weighted pass rates. Its strict and relaxed thresholds differ in whether they tolerate missed criteria, including potentially decisive ones.
4.1 Harness
- Each task uses an OpenHands agent loop in a Harbor container with document tools, bounded turns and time, and network access limited to the model provider’s API.The environment includes document-processing software, but no web search or package installation.Source: §4.1 · p. 5
4.2 Models and attempts
- Thirty configurations from 13 developers attempt every task five times; rate-limit errors can trigger up to three reruns, while failed attempts are excluded from task means.The configurations come from the September 2026 launch leaderboard.Source: §4.2 · p. 5
4.3 Grading
- An OpenCode agentic judge inspects the workspace and trajectory, can run code, and returns a binary verdict with reasoning and file evidence for every criterion.It is instructed to recompute values and inspect rendered outputs rather than trust formulas, scripts, or unsupported final-message claims.Source: §4.3 · p. 5
4.4 Metrics
- Strict pass@1 requires meeting every criterion, while relaxed thresholds accept some misses; criteria are unweighted, so a 95% pass can miss a decisive release criterion.Tasks are weighted equally within each domain; the leaderboard reports strict pass@1 for 50 healthcare and 80 finance tasks across five attempts per task.Source: §4.4 · p. 5, Table 3 · p. 6
5 Results
Strict pass@1 is low across configurations, with the leader ahead in both domains and configuration rankings broadly similar. Relaxing the rubric threshold raises measured pass rates, while the leader also sits at the end of the cost frontier.
5.1 Main results
- Claude Opus 5.5 (adaptive/max) leads strict pass@1 with 24.7% in healthcare and 23.9% in finance; the median configuration scores 0.6% and 2.5%, respectively.In healthcare, GPT-6 Astra (max) is next at 11.6%, 13.1 points lower.Source: §5.1 · p. 6
- Among configurations scoring above zero in either domain, strict-pass rankings are similar across healthcare and finance (Spearman ρ = 0.82).Scores are higher in finance for 21 configurations and higher in healthcare for two.Source: §5.1 · p. 6
5.2 Relaxed thresholds
- At 95%, the top five configurations’ scores multiply by 2.7 in healthcare and 1.8 in finance on average; pass@1 (N−1) gives the leader 37.8% and 32.9%.The leader ranks first at every threshold, and 95%-threshold rankings agree closely with strict rankings (Kendall τb = 0.85 in healthcare and 0.92 in finance).Source: §5.2 · pp. 6–7
- At the 90% threshold, the leader reaches 62.4% in healthcare and 58.3% in finance, while the median reaches 6.4% and 10.4%.Figure 2 contrasts strict pass@1 by domain with relaxed-threshold pass@1 for the leader and median.Source: §5.2 · p. 6, Figure 2 · p. 7
5.3 Cost and tokens
- The cost frontier ends at Claude Opus 5.5 (adaptive/max), costing $7.10 per healthcare attempt and $11.47 per finance attempt.The frontier also includes GPT-6 Astra (max) and Muse Spark 1.3 (max) at lower cost; the most expensive configuration, Claude Fable 5.1 (adaptive/max), is not on it. The leader processes 9.8 million tokens per healthcare attempt and 17.8 million per finance attempt, 94% and 95% cached input.Source: Figure 3 · p. 7, §5.3 · p. 7
6 Case studies
The case studies show agents following requests whose premises conflict with the record and producing internally consistent work from incorrect inputs.
Unchallenged premises
- In two healthcare cases, agents accepted the requested diagnosis or treatment plan despite chart details that called for urgent evaluation or caution about debridement.The rubric required challenging the Bell’s palsy diagnosis, withholding prednisone pending exclusion of acute ischemia, and recommending urgent stroke-capable evaluation; another case required raising pyoderma gangrenosum and advising against debridement pending biopsy or specialist evaluation.Source: §6 · p. 8
Wrong inputs in consistent analyses
- In finance cases, incorrect prices, sales adjustments, or check counts led to flawed release, leverage, or outsourcing decisions despite coherent analyses.The examples include a $25.7 million overstatement left in a fund report, a debt-funded buildout that breaches a covenant when signs are corrected, and a 2025 check count of 2,838 instead of 138.Source: §6 · p. 8
7 Conclusion
DAYJOB evaluates whether agents can turn realistic professional requests and workspaces into deliverables that satisfy expert rubrics. Results remain low, and the authors release the benchmark materials while planning expansion to further professional domains.
7 Conclusion
- DAYJOB’s 130 healthcare and finance tasks test whether agents can identify the task, find relevant facts, and detect when a request’s premise is wrong.The tasks were built from the work of professionals in both domains and require deliverables to meet an expert rubric in full.Source: §7 · p. 8
- The strongest configuration passes 24.7% of healthcare and 23.9% of finance attempts, while 19 configurations score below 5% in both domains.Case studies show agents accepting premises contradicted by the record and carrying wrong inputs through otherwise consistent analyses; criterion-level judge verdicts identify missed facts or decisions.Source: §7 · p. 8
- The release includes all healthcare tasks, 50 of 80 finance tasks, the evaluation harness, and the leaderboard; the authors plan to extend DAYJOB to further professional domains.Source: §7 · p. 8
A Strict pass@1 by configuration
Figure 4 plots strict pass@1 for all 30 configurations in Table 3 order.
A Strict pass@1 by configuration
- Figure 4 shows strict pass@1 for all 30 configurations, ordered as in Table 3.Source: Figure 4 · p. 11