PaperlayerView exact revision on arXiv

Source-linked AI summary

HCAST: Human-Calibrated Autonomy Software Tasks

David Rein, Joel Becker, Amy Deng, Seraphina Nix, Chris Canal, Daniel O'Connel, Pip Arnott, Ryan Bloom, Thomas Broadley, Katharyn Garcia, Brian Goodrich, Max Hasin, Sami Jawhar, Megan Kinniment, Thomas Kwa, Aron Lajko, Nate Rush, Lucas Jun Koba Sato, Sydney Von Arx, Ben West, Lawrence Chan, Elizabeth Barnes

arXiv:2503.17354v1cs.AI

TL;DR

HCAST addresses the gap between benchmarks of AI expertise and evidence about autonomous real-world competence. It combines a 189-task, four-domain benchmark with human task-time baselines. The best models succeed 70–80% of the time on tasks taking humans less than one hour, but less than 20% on tasks taking humans more than four hours.

  • Problem

    Expert-level benchmarks capture knowledge and reasoning but are not necessarily predictive of real-world competence, motivating benchmarks grounded in effects relevant to autonomous AI systems.

    Source: §1 · pp. 1–2

  • Method

    HCAST combines 189 tasks across four domains with 563 attempts by skilled people to estimate human task times and contextualize agent performance.

    Source: §1 · p. 2

  • Results

    70–80% of the time on tasks taking humans less than one hour, but less than 20% on tasks taking humans more than four hours, the best models succeed.

    Source: §1 · p. 4

  • Takeaways & Limitations

    HCAST focuses on human task times and models’ corresponding success rates across tasks of different lengths.

    Source: §6 · p. 15

    Human baselines are slow and expensive to collect; 79 tasks have no successful human baseline.

    Source: §5.3 · p. 15

Abstract

from arXiv · show

To understand and predict the societal impacts of highly autonomous AI systems, we need benchmarks with grounding, i.e., metrics that directly connect AI performance to real-world effects we care about. We present HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark of 189 machine learning engineering, cybersecurity, software engineering, and general reasoning tasks. We collect 563 human baselines (totaling over 1500 hours) from people skilled in these domains, working under identical conditions as AI agents, which lets us estimate that HCAST tasks take humans between one minute and 8+ hours. Measuring the time tasks take for humans provides an intuitive metric for evaluating AI capabilities, helping answer the question "can an agent be trusted to complete a task that would take a human X hours?" We evaluate the success rates of AI agents built on frontier foundation models, and we find that current agents succeed 70-80% of the time on tasks that take humans less than one hour, and less than 20% of the time on tasks that take humans more than 4 hours.

1 Introduction

Benchmarks of AI expertise do not necessarily measure autonomous real-world competence, while existing agent benchmarks may be narrow or lack directly comparable human performance. HCAST addresses this with diverse, realistic tasks and human time baselines, and its agents’ success rates fall as estimated human task time increases.

Benchmark desiderata

  • The benchmark aims for open-ended problems, diverse tasks, and action in real software environments with feedback to address concerns about memorization, narrow metrics, and realism.These are stated desiderata for benchmarks intended to assess autonomous capabilities.Source: §1 · p. 1

Motivation

  • Existing expert-level question benchmarks capture knowledge and domain reasoning but are not necessarily predictive of real-world competence; general agent benchmarks may be narrow or lack direct human comparisons.Some real-task benchmarks rely on informal time estimates or compare humans and AI under different conditions.Source: §1 · pp. 1–2

Benchmark

  • HCAST comprises 189 tasks across four domains, designed to reflect realistic, varied-complexity work that can take from 1–2 minutes to 8+ hours.Tasks typically involve multi-step decisions, derive largely from real-world work, and undergo multiple stages of manual quality assurance and review.Source: §1 · p. 2

Human baselines

  • To ground agent scores in human task time, 140 skilled people made 563 attempts, enabling analysis of the time humans take on tasks agents complete.The introduction frames these human baselines as a way to draw more direct conclusions from agent scores.Source: §1 · pp. 2, 4

Released examples

  • The authors release 11 example task families and withhold the rest to reduce data contamination and hill-climbing against HCAST.The examples span the four primary domains.Source: Figure 2 · p. 3, §1 · p. 4

Main findings

  • 70–80% of the time on tasks taking humans less than one hour, but less than 20% on tasks taking humans more than four hours, best models succeed.The authors also report that successful agents use 5–15 actions on a third of tasks, while roughly 10% average above 25 actions.Source: §1 · p. 4

2 Benchmark Creation Process

HCAST was built through task creation, iterative review, and human baselining. Its tasks cover varied real-world activities and are specified with environments and scoring, while quality assurance and human-baseline procedures aim to improve reliability and comparability.

2 Benchmark Creation Process

  • The team created 189 tasks, applied multi-stage manual quality assurance with many revisions, and measured how long skilled humans took to complete them.Tasks came from internal creation and a contributor bounty, followed by human and agent test runs and review.Source: §2 · p. 4, §2.1 · p. 4

2.1 Task Desiderata

  • Tasks represent end-to-end human activities across varied lengths and skills, emphasizing open-ended work, code-based tasks, and economically valuable capabilities.The authors focus less on vision-dependent tasks and expect this selection to track real-world impact more closely than tasks AI systems are not optimized to perform.Source: §2.1 · p. 4

2.2 Task Structure

  • Each task has instructions, a containerized environment, and a scoring function; most scores are binary, while continuous scores are binarized using a heuristic success threshold.Tasks are grouped into 78 families sharing setup code and scoring logic, though family tasks are created individually.Source: §2.2 · p. 5

2.3 Quality Assurance (QA) and Review

  • Quality assurance includes fresh human testers, revision checks with gold solutions where feasible, and manual review of five AI-agent runs per task.Reviewers inspect 945 transcripts to check for bugs, reward hacking, unfair failures, or other task issues.Source: §2.3 · pp. 5–6

2.4 Human Baselines

  • Qualified baseliners with relevant technical backgrounds attempt tasks using a platform and tools intended to support fair comparisons and representative workflows.Humans receive the same task environment and instructions as agents, use the environment via SSH, and most baselines used for time estimates are manually reviewed.Source: §2.4 · p. 6

3 Benchmark Analysis

HCAST spans 189 tasks across four domains, with task difficulty and structure varying widely. The benchmark estimates human completion times using successful baselines or forecasts and categorizes tasks by contamination risk.

3.1 Qualitative Descriptions of Tasks

  • Task families differ in how much their tasks vary, from similar research questions to diverse CTF challenges or versions offering different amounts of help.Source: §3.1 · p. 8
  • Tasks vary in required expertise, openness, assistance, feedback, and realism, ranging from exploration-heavy debugging to tasks that penalize exploration or trade realism for targeted measurement.Source: §3.1 · pp. 7–8

3.2 Human Time Estimates

  • Of 563 human attempts, 341 succeeded, and 110 of 189 tasks had at least one successful baseline.Source: §3.2 · p. 9
  • Task time estimates use the geometric mean of successful baseline times, or a creator or knowledgeable author’s forecast when no baseline succeeds.Source: §3.2 · p. 9
  • R² = 0.51 links forecasted task length with successful-baseliner time, while forecasts account for a higher proportion of estimates on longer tasks because baselines are logistically harder to collect.Source: Figure 3 · p. 8, §3.2 · p. 9

3.3 Contamination Risk

  • HCAST distinguishes mostly private solutions, public solutions requiring substantial adaptation, and short solutions that may be readily memorized; these labels rely on author judgment and may be mistaken.Source: §3.3 · p. 10

4 Agent Performance

Agent success declines with estimated task difficulty, while Claude 3.7 Sonnet and o1 maintain the highest success rates across difficulty levels. Successful task completion often involves multiple environment actions, but the reported counts depend on the agents used.

4.1 Agent Scaffolding

  • The Modular and Triframe scaffolds use sequential tool interactions, with Triframe selecting a top-scoring command from six proposed actions.GPT-4o and Claude 3.5 Sonnet use Modular; o1 and Claude 3.7 Sonnet use Triframe.Source: §4.1 · p. 11

4.2 Results

  • Agent performance decreases as estimated task difficulty rises, with a significant drop from the 15-minute-to-1-hour bucket to the 1-to-4-hour bucket.Claude 3.7 Sonnet and o1 maintain the highest success rates across all difficulty levels, suggesting reasoning capabilities may advantage complex-task completion.Source: §4.2 · pp. 11–12, Figure 6 · p. 12

4.3 Number of Actions

  • For roughly a third of tasks, successful agents averaged 5–15 actions, while roughly 10% averaged more than 25.An action is Python or Bash code execution; internal thinking and reflection are excluded.Source: §4.3 · p. 12
  • The action counts do not establish the minimum steps a task requires because they depend on the specific agents evaluated.A more capable agent might solve the same tasks in fewer steps.Source: §4.3 · p. 12

5 Discussion

The discussion highlights limits in task realism, interpreting human baselines, and the practical cost of maintaining HCAST and collecting baseline attempts.

5 Discussion

  • HCAST’s task realism, interpretation of human time estimates, and operational costs constrain how directly its results predict autonomous AI impacts.The discussion identifies these as continuing challenges despite HCAST’s diverse tasks and human-expert difficulty estimates.Source: §5 · p. 12

5.1 Challenges resulting from limits in task realism

  • HCAST tasks have bounded scopes, algorithmic scoring, and no required human interaction, unlike many real-world work settings.The authors also note that task families vary in economic relevance, making performance levels difficult to interpret as impacts.Source: §5.1 · pp. 13–14

5.2 Challenges with interpreting human baseline results

  • Human time estimates may be biased upward because baseliners start with little prior context and spend time getting up to speed.The authors say this may differ from tasks people complete in their real work.Source: §5.2 · p. 14
  • Harder tasks were preferentially assigned to baseliners judged more likely to succeed, potentially shifting time estimates for easier versus harder tasks.Matching baseliners to suitable expertise was occasionally imperfect.Source: §5.2 · p. 14
  • Only 61% of human baselines succeeded, and those rates are not directly comparable to agent success rates because motivation and environmental setup differ.The authors suggest performance-based compensation may encourage quick completion and task abandonment.Source: §5.2 · p. 14

5.3 Additional limitations

  • HCAST’s complex environments make evaluations expensive and difficult to validate, run, and maintain, limiting fine-grained statistically significant comparisons between agents.Some tasks require GPUs, auxiliary virtual machines, internet resources, or external APIs.Source: §5.3 · pp. 14–15
  • Human baselines are slow and expensive to collect; 79 tasks had no successful human baseline, including 50 with no collected attempts.Small timing-software errors may remain, and some large errors may remain as well.Source: §5.3 · p. 15

6 Conclusion

HCAST links realistic task duration for skilled humans to AI agents’ success across task lengths, using manually refined tasks and human baselines. Earlier HCAST versions were also used in pre-deployment evaluations that uncovered task errors and validated relevance to broad capabilities.

6 Conclusion

  • HCAST focuses on measuring how long realistic tasks take humans and how agent success varies with those task lengths, rather than on tasks that are especially difficult to complete.The benchmark covers machine learning, cybersecurity, software engineering, and general reasoning.Source: §6 · p. 15
  • The authors refined and quality-checked tasks, collected human baselines to ground agent performance in completion time, and reported results for four frontier AI models.Source: §6 · p. 15
  • Earlier HCAST versions were used in pre-deployment evaluations that uncovered task errors and validated the benchmark’s relevance to broad capabilities.The evaluations were conducted by METR.Source: §6 · p. 15
  • Anchoring agent capabilities to human time estimates offers an intuitive metric for understanding autonomous systems’ real-world impact across economically valuable domains.Source: §6 · p. 15

A Author Contributions

  • David Rein managed task collection, QA/testing, and baselining and wrote most of the paper; Joel Becker led human-baseline collection and infrastructure development.Source: A Author Contributions · p. 18

C Related Work

Prior benchmarks cover expert knowledge, general agent skills, and domain-specific tasks, but often lack long-horizon realism or direct human comparisons. The additional analyses examine continuous scores and an alternative weighting across task families.

C Related Work

  • Question-answer benchmarks can test substantial domain expertise without requiring much time from experts, and may not capture the skills needed for longer-horizon tasks.The paper notes that foundation models may benefit from memorized specialized knowledge.Source: C Related Work · pp. 18–19
  • General agent benchmarks may be hard to interpret when they lack direct human comparisons or use small tasks unlike economically valuable human work.Source: C Related Work · p. 19
  • Benchmarks span HCAST’s domains, including ML and software engineering tasks, cybersecurity exercises, and text-based scientific research environments.Source: C Related Work · p. 19
  • Direct human comparisons are costly, and competition-based human percentiles may not reveal the time and resources humans spent on the tasks.The authors say they invested thousands of human hours in HCAST QA and baselines.Source: C Related Work · p. 19
  • Creating tasks from scratch and withholding most HCAST tasks are presented as ways to mitigate, but not eliminate, data-contamination concerns.Source: C Related Work · pp. 19–20

D Additional Analyses

  • Figure 8 presents average agent scores by task length; continuous scores are typically higher than binary success rates because they capture partial progress.Source: Figure 8 · p. 20, D Additional Analyses · p. 20
  • The alternative weighting multiplies each task’s score or success rate by the inverse square root of its family’s task count before averaging.Source: D Additional Analyses · p. 20

Premature Task Abandonment

Agents sometimes abandon challenging problems despite having computational resources left, submitting answers they have already verified as incorrect.

Premature Task Abandonment

Error Recovery and Strategic Adaptation

Agents still struggle to change course when an approach stalls, and limited-feedback tasks pose particular difficulty. The section describes repeated failed strategies, poor resource allocation, and low scores on blackbox optimization tasks.

Error Recovery and Strategic Adaptation

  • Models have improved at recovering from mistakes and avoiding loops, but still struggle to course-correct when their approach is not making progress.In eliciting secret, they often persisted with failed approaches, restarting conversations with different prompts instead of continuing a conversation that could yield more information.Source: Error Recovery and Strategic Adaptation · pp. 20, 22
  • Figures 9 and 10 show average scores and success rates across task lengths using inverse-square-root weighting by task-family size.Source: Figure 9 · p. 21, Figure 10 · p. 21
  • With about 1.57 million of 2 million tokens used, Claude 3.5 Sonnet submitted a known incorrect answer instead of trying to reach a final answer.The scaffold had reminded the model to submit soon.Source: Error Recovery and Strategic Adaptation · p. 22

Navigating Low-Feedback Environments

  • 0.2144 and 0.2639 are the average agent scores on blackbox optimize tasks in the <15min and 15min-1h difficulty buckets, respectively.These tasks limit attempts to optimize a blackbox function without cheap verification.Source: Navigating Low-Feedback Environments · p. 22

E.1 Baselining

Baselining quality was checked through recorded work and manual review, with corrections and targeted inspection addressing reported or anomalous scores and times.

E.1 Baselining

  • Baseliners generally could not complete multiple tasks from one family, with exceptions for families judged diverse enough to avoid an expected learning effect.Source: §E.1 · p. 22
  • Baseliners recorded work and verbalized their process, allowing checks on protocol adherence and qualitative review of success and failure.Source: §E.1 · p. 22
  • Manual review implemented 30% of proposed score changes and 71% of reported clock-time changes.Source: §E.1 · p. 22
  • The team manually reviewed most successful baselines, inspecting notes, recordings, and transcripts to verify scores and clock times and correct mistakes.Reviews prioritized unusually short or long times and results that disagreed with other baselines.Source: §E.1 · p. 22
  • Setup issues were minimal overall and concentrated in qualification tasks; 19 sampled debug-small-libs qualification attempts averaged 9 minutes of setup issues versus 1.2 hours total task time.The median setup-issue time was 6 minutes.Source: §E.1 · p. 22
  • Some early baselines lacked screen recordings; although performance incentives could encourage cheating, reviewers observed no explicit cheating in recordings.Source: §E.1 · p. 22

E.2 Qualification tasks

Baseliners generally qualified in the relevant domain before attempting tasks, with additional expertise checks for two cryptography tasks. Incentives rewarded successful completion and faster performance.

E.2 Qualification tasks

  • Most baseliners had to pass domain-specific qualifications before attempting tasks; general-reasoning qualification could be met through any domain’s qualification.Source: §E.2 · p. 23
  • Qualification tasks checked domain competence rather than advanced specialization and were easier than many tasks in their domains.Source: §E.2 · p. 23
  • Spn cryptanalysis and hash collision baseliners also needed demonstrated cryptography expertise beyond cybersecurity qualification.Source: §E.2 · p. 23

E.2.1 Incentives

  • Successful completion earned a bonus proportional to the task’s average human time and the baseliner’s hourly rate, encouraging success on tasks of widely varying lengths.The passage gives the bonus as 2 ∗ t, where t is average task time and r is the hourly rate.Source: §E.2.1 · p. 23
  • A separate speed bonus rewarded the fastest successful baseliner, aiming to limit prolonged rechecking and push time estimates toward a lower bound.For tasks without successful attempts, the speed bonus was shared among those who attempted and failed.Source: §E.2.1 · p. 23
  • A small number of early baselines without these incentives were retained because they still provided useful data and their time estimates appeared reasonable.Source: §E.2.1 · p. 23

F Task Summaries

This section catalogs 78 task families, providing task metadata and concise descriptions, and documents how those descriptions were generated. It also covers software, machine-learning, and reasoning tasks with human-time baselines or estimates of 8.0–30.0 hours, their reported run outcomes, and the fact that many of the longest tasks have no successful runs reported.

F Task Summaries

  • The table summarizes 78 task families with IDs, geometric-mean time estimates, majority estimate sources, subdomains, contamination and expertise levels, and action-count statistics.Action counts are geometric means across successful agent runs, then arithmetically averaged across tasks in each family.Source: F Task Summaries · p. 23
  • Family descriptions were generated by Claude 3.7 Sonnet using task code, tracked metadata truncated to 10,000 characters, author-provided information, five few-shot examples, and summary-writing instructions.Source: F Task Summaries · pp. 23–24
  • 1m and 8.0h examples show the catalog spans simple Wikipedia research and few-shot prompting strategy development.Source: F Task Summaries · pp. 24, 30
  • 8.0h cuda backtesting and 8.5h mlab are the shortest tasks listed in this part, with the former reporting no successful runs.CUDA backtesting aims for a 30x speedup using custom kernels while preserving functionality; mlab consists of machine-learning bootcamp exercises with algorithm implementations and unit tests.Source: F Task Summaries · pp. 30–31
  • 10.7h continue pattern, 10.0h gpt2 algo circuits, 8.6h number list steganography, and 11.2h ICLR authors span reasoning, machine learning, and software implementation.The listed tasks predict a bitstring’s next 10,000 characters, train a tiny transformer for shortest-path computation, conceal messages in number lists, and scrape ICLR 2023 author names and emails into a sourced CSV; the first two are reported with no successful runs.Source: F Task Summaries · p. 31
  • acdc bug is 14.6h (baseline) with 39.0 actions; ACE-OAuth is estimated at 13.0h and backdoor image classifier at 11.0h, both with no successful runs; pruning attention heads is estimated at 11.0h.Source: F Task Summaries · pp. 31–32
  • 30.0h robot control, 20.0h lie detector, 15.4h gradient inversion, and 14.5h UAV search are reported with no successful runs.The tasks involve spline-following control despite wheel slippage and motor limits, reducing a truth-detection probe’s accuracy while retaining benchmark performance, reconstructing images from federated-learning gradients, and locating target structures on a simulated map.Source: F Task Summaries · pp. 31–32
Loading 2503.17354v1…