PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring AI Ability to Complete Long Software Tasks

Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan

arXiv:2503.14499v4cs.AIcs.LG

TL;DR

Benchmark performance does not clearly indicate real-world AI capability, motivating a human-grounded measure of task difficulty. The paper defines and evaluates a 50% task-completion time horizon using expert baselines and software-related tasks, finding exponential growth from 2019–2025. If this trend generalizes to real-world software work, extrapolation places one-month task automation around 2028–2030.

  • Problem

    Existing benchmarks provide limited general, intuitive, and quantitative evidence for comparing AI capabilities across models and real-world task difficulty.

  • Method

    The paper combines HCAST, RE-Bench, and 66 SWAA tasks with expert human timing, evaluates frontier models, and estimates the task duration corresponding to 50% success.

  • Results

    The 50% task-completion time horizon grew exponentially from 2019–2025, with a doubling time of approximately seven months.

  • Takeaways & Limitations

    If observed trends generalize to real-world software tasks, an 80% confidence interval places AI capable of completing one-month tasks between mid-2028 and mid-2030.

  • Takeaways & Limitations

    The task suite differs from realistic work involving other agents, resource constraints, dynamic environments, and high reliability requirements, limiting confidence that results generalize.

Abstract

from arXiv · show

Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024. The increase in AI models' time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes, combined with better logical reasoning and tool use capabilities. We discuss the limitations of our results -- including their degree of external validity -- and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.

1 Introduction

The paper introduces task-completion time horizon as an intuitive human-grounded measure of AI capability, addressing benchmark limitations. Across 2019–2025 evaluations, the 50% horizon grew exponentially, while extrapolation remains constrained by external validity.

  • Motivation: Existing benchmarks are often artificial, adversarially selected, rapidly saturating, and difficult to compare across models with vastly different capabilities.These limitations make general AI capability progress difficult to interpret.
  • Contribution: The task-completion time horizon measures the duration of tasks models complete at a specified success probability, operationalized here as the 50% time horizon.It relates AI performance to the time human experts typically require for the same tasks.
  • Implications and limitations: Extrapolation implies a time horizon above one month between mid-2028 and mid-2031, but this depends on external validity and future changes in the trend.The evaluated tasks do not perfectly represent researchers’ and software engineers’ broader intellectual labor.

2 Measuring AI agent performance on realistic tasks

The study evaluates AI agents on software and machine-learning tasks spanning seconds to 30 hours, using human baselines and task-weighted model success rates. Success generally declines as human task time increases, while newer models complete longer tasks.

  • Task suites: The suite contains 97 HCAST software tasks, 7 eight-hour RE-Bench tasks, and 66 SWAA atomic actions lasting 1 second to 30 seconds.Tasks span approximately 1 minute to 30 hours in HCAST, while RE-Bench tasks are all eight hours long.
  • Task suites: Tasks are automatically scored using either a continuous score or a binary threshold.The paper normalizes and processes these scores for analysis.
  • Baselining: Human baselining uses over 800 attempts totaling 2,529 hours, with 148 of 169 tasks covered by human baselines.The baseliners are skilled professionals in software engineering, machine learning, and cybersecurity, averaging about five years of relevant experience.
  • Results: Recent models completed approximately 50% of all tasks, whereas earlier models performed substantially worse; task-completion patterns correlated at approximately 0.73 on average.Results report 12 frontier and 4 near-frontier models released between 2019 and 2025, with eight runs per agent/task pair.
  • Results: Across models, success rates decline as human completion time increases, and this relationship is well fit by an exponential model with R2 ≈0.80.Recent models including Claude 3.7 Sonnet and o3 completed some tasks taking human baseliners more than four hours.

3 Time horizon

The paper converts model task performance into a human-time horizon using binary success labels and logistic regression, then examines its trend across model release dates. Horizon length increased substantially from 2019 to early 2025, with recent models reaching much longer tasks, while failures reveal improved adaptation but persistent weaknesses.

  • 3.1 Computing time horizon: Binary task success is derived from naturally binary tasks or task-specific thresholds for continuously scored tasks.HCAST uses the human baseliner’s target score, while RE-Bench uses the average score of 7–9-hour human runs.
  • 3.1 Computing time horizon: The logistic model estimates the task difficulty at which each agent has a 50% success probability, using human baseline time as task difficulty.The learned parameter h_agent represents the 50% time horizon.
  • 3.2 Time horizon vs. release date: 207 days: the fitted time horizon doubling interval, with a 95% bootstrapped confidence interval of 166–240 days.The regression relates log(time horizon) to model release date using a three-level hierarchical bootstrap for error bars.
  • 3.2 Time horizon vs. release date: GPT-2 has a 50% time horizon of 2 seconds, whereas o3 reaches 110 minutes and succeeds on several tasks exceeding 4 hours.o3 lies above the long-run trend, with p = 0.006, possibly indicating faster growth in 2024 and early 2025.
  • 3.2 Time horizon vs. release date: 80% time horizons double on a similar schedule—204 days versus 207 days for 50% horizons—but are 4–6x shorter.The shorter 80% horizons indicate that reliable success remains limited to substantially shorter tasks.

4 External validity and robustness

The authors test whether the observed exponential trend survives alternative benchmarks and more realistic task settings. The trend persists, but task messiness and human time-rating choices affect measured horizons and leave external validity unresolved.

  • Robustness checks: The authors perform a retrodiction check without SWAA and conduct three experiments addressing external validity.These checks compare the 2023–2025 trend with the trend since 2019 and examine alternative task settings.
  • SWE-bench Verified: The exponential trend holds on SWE-bench Verified, although its estimated doubling time is shorter than on the main task suite.SWE-bench Verified consists of automatically checkable, well-specified software issues from large open-source repositories.
  • Task messiness: Models perform worse on tasks with higher messiness scores after controlling for task length, although performance trends are similar across messiness levels.Messiness includes resource limits, novelty, and dynamic environments.
  • Internal pull requests: Internal pull-request results align better with contractor than maintainer completion times, while agent performance is worse than maintainer-time predictions.Contractors took 5–18x longer than repository maintainers to fix the issues.
  • SWE-bench Verified: 70 days: the SWE-bench Verified doubling time, compared with 143 days for HCAST + SWAA + RE-Bench using 2024 models.The difference may reflect SWE-bench difficulty annotations underestimating contractor completion times for easier tasks.

5 Extrapolation

The paper extrapolates the measured horizon trend to software tasks lasting one month, while emphasizing that the forecast depends on trend continuation and generalization beyond the evaluated task distribution. Sensitivity analysis identifies methodological noise, external validity, and future trend changes as major uncertainties.

  • Extrapolating towards one-month-horizon AI: A one-month software-task horizon is defined as approximately 167 working hours, chosen to compare AI work capacity with human working time.The extrapolation concerns software tasks rather than one-month AGI.
  • Sensitivity analysis: The forecast varies under perturbations of tasks, runs, models, weighting, regularization, and baseline-time noise.The sensitivity analysis applies 10,000 random perturbations and includes multiple hyperparameter combinations.
  • Forecast uncertainty: External validity concerns and future changes in the growth trend account for most of the forecast uncertainty not represented by the sensitivity plot.Real forecasts are expected to have larger error than the naive extrapolation because predicting the future is intrinsically difficult.
  • Forecast uncertainty: Mid-2029: the central estimate has an approximately two-year 80% confidence-interval width, under the main analysis assumptions.If the faster 2024–2025 trend continues, half the probability falls in 2027 and potentially late 2026.
  • Task distribution limitations: Realistic tasks often lack automatic scoring and involve other agents, resource constraints, dynamic environments, and high reliability requirements.These differences cast doubt on whether benchmark improvements generalize to real-world tasks.

6 Discussion

The paper presents task-completion time horizon as a human-grounded metric and reports rapid exponential growth on the evaluated software and research tasks. It cautions that forecasts and measurements remain domain- and task-distribution-dependent, especially when extending them to real-world work.

  • 6 Discussion: The task-completion time horizon relates AI performance to the typical time human experts require to complete tasks.The study combines 66 new SWAA tasks with existing benchmarks and human baselines.
  • 6 Discussion: A seven-month doubling time describes 50% task-completion horizon growth from 2019–2025 on the evaluated tasks.Current systems still perform worse on less structured, messier tasks.
  • 6 Discussion: Mid-2028 to mid-2030: the stated 80% confidence interval for AI completing one-month software tasks, conditional on generalization and continued trends.The forecast could be as soon as early 2027 if the 2024–2025 trend continues.
  • 6 Discussion: Time horizon depends on the domain, task distribution, and human baseliners’ skill and context, and becomes harder to measure for longer horizons or success rates near 100%.The authors identify multiple domains, more models, improved elicitation, and alternative statistical methods as future work.

A.1 Agent and capability benchmarks

Recent agent benchmarks evaluate dynamic, multi-step capabilities, but they lack a unified metric for comparing models across capability levels. This work uses task duration and structured task suites to support cross-benchmark progress measurement.

  • Agent benchmarks increasingly test dynamic behavior across web browsing, coding, games, and other environments.
  • Software benchmarks span programming challenges, complex software tasks, and research engineering tasks, including SWE-bench Verified and RE-Bench.
  • A unified metric is needed to track progress over time and compare models with vastly different capabilities across benchmarks.
  • The study’s task suites isolate reliably scorable units of work, typically requiring less context than tasks embedded in larger projects.
  • HCAST contains 97 tasks across 46 families covering cybersecurity, machine learning, software engineering, and general reasoning.
  • SWAA adds five families of shorter software actions, including common decisions, code completion, and mathematics, to measure earlier models.

B.2 Limitations of the task suite

The task suite differs from realistic work through automatic scoring, limited interaction and resources, static environments, and low penalties for mistakes. These differences constrain external validity, although messier tasks show lower absolute performance but similar trends.

  • All tasks use automatic scoring, constraining solution formats, open-endedness, and the need for sensible value judgments.
  • No task involves interaction with other autonomous agents, omitting strategic decision-making, real-time coordination, and prediction of other agents.
  • Few tasks impose salient resource constraints, whereas real-world work often requires efficient use of limited resources.
  • Very few tasks punish individual mistakes, unlike real-world settings where a single blunder can substantially reduce success.
  • Tasks typically use static environments, while real tasks often occur amid changing conditions.
  • Messier tasks have lower absolute model performance, but performance trends resemble those on less messy tasks.
  • Conditioning human difficulty ratings on successful runs can shorten estimated task lengths and may underestimate the pace of model improvement.
  • Human time-horizon estimates are around 1.5 hours rather than the expected 8 hours, likely reflecting failures associated with the incentive scheme.

C.2 Internal PR tasks

The internal PR experiment compared AI agents, external baseliners, and repository maintainers on five real software issues, revealing that context strongly affects human completion time and model performance does not track maintainer time straightforwardly.

  • Human baselines: 5x-18x longer: contract baseliners took substantially more time to resolve issues than repository maintainers.The comparison suggests the measured horizon better corresponds to low-context human labor than to high-context maintainer labor.
  • Method: Five recent, uncontaminated internal repository issues were evaluated with GPT-4o, Claude 3.5 Sonnet (New), and o1, alongside human baseliners and maintainers.Maintainers manually scored model and baseliner solutions using pull-request criteria.
  • Evaluation: Repository maintainers scored results more consistently than contractors, with correlations of 88–91% versus 50–60%.Maintainer scoring was used to assess whether solutions could be merged as pull requests.
  • Model–human mismatch: Baseliner success rates were reasonably consistent with model success curves when contractor completion time measured task length, despite baseliners taking much longer than maintainers.This supports interpreting the time horizon relative to the context level of the human baseline.
  • Model–human mismatch: Issue 11 took maintainers less than five minutes but was never completed successfully by models in 30 runs, showing maintainer time poorly predicted model performance.The task required adding simple comments across 10 Python files.
  • Evaluation: The evaluation used common agent scaffolds and human-equivalent tool affordances, with no task-specific prompting except a simple SWAA scaffold.GPT-2 received imputed zero scores because it was incompatible with the scaffolding.

D.2.1 What do newer models do better?

Newer models improve at logical and programmatic problem solving, tool use, and recovering from mistakes, but still struggle with messy tasks, unclear feedback, and proactively seeking information.

  • Robustness to mistakes: Greater robustness to mistakes lets newer models recover and course-correct instead of looping or introducing additional errors.The paper highlights this improvement particularly on harder coding and capture-the-flag tasks.
  • Tool use: Later models use available tools more reliably, while earlier models often struggle with tool-call syntax, file editing, or inefficient command sequences.o1-preview consistently edited files efficiently in the cited example, whereas GPT-3 and GPT-4o showed formatting or command-use problems.
  • Logical and programmatic problem solving: Improved logical and programmatic problem solving increases models’ ability to interpret instructions and write syntactically and semantically correct code.This supports more reliable use of code to find correct answers.
  • Remaining limitations: Models still perform worse on qualitatively messier tasks and often cannot judge correctness without clear feedback or cheap verification.Blackbox and symbolic-regression tasks were difficult because probing information was costly and success was hard to verify.
  • Remaining limitations: Agents often assume they know how to proceed rather than proactively consulting relevant information, such as available API documentation.They typically read documentation only after environmental errors expose their initial guesses.
  • Remaining limitations: An additional encoding example shows that even strong models tried multiple Python encodings instead of using the bash file command to identify the correct one.The resulting trial-and-error was inefficient and wasted tokens.

E.4 Limitations and future work

The paper identifies limitations in model elicitation, human baselines, task realism, inference-time computation, and statistical analysis, and proposes targeted improvements.

  • Model elicitation: Limited elicitation effort means the reported results are reasonable lower bounds, since some models may have greater capabilities than demonstrated.The authors devoted substantially more elicitation effort to o1 and the original Claude 3.5 Sonnet than to most other models.
  • Human baselines: Human task-time estimates may be noisy or systematically skewed because successful completions are selected, participants may give up, and baseliner skills vary.The authors call for more rigorous human baselining.
  • Task realism: The task distribution may differ systematically from economically valuable or risk-relevant work, including through its narrow interaction modality and omission of mouse-based tasks.The authors recommend more natural and varied tasks.
  • Inference-time computation: Limited inference-time computation leaves room to improve performance while retaining cost advantages over human work.At an assumed human compensation of $143.61/hour, more than 80% of successful runs cost less than 10% of comparable human labor.
  • Data analysis: The apparent 2024–2025 acceleration is uncertain because few models support the comparison, and the analysis loses information across multiple estimation stages.Future work should test a possible slope change and construct more credible forecast intervals.

F.1 Retrodiction from 2023–2025 data

The authors extend the time-horizon trend backward with SWAA, finding a roughly seven-month-scale doubling over 2019–2025 while noting that recent growth may be faster but remains uncertain.

  • Trend estimates: Exploratory 2023–2024 data suggested time horizons doubled about every six months, but the estimate had wide error bars because only two 2023 models were available.The analysis covered nine frontier and near-frontier models using HCAST and RE-Bench.
  • Retrodiction method: SWAA reduced the shortest human task duration from 1 minute to under 2 seconds, enabling measurement of GPT-2, davinci-002, and GPT-3.5-turbo-instruct.The expanded suite combined SWAA with HCAST and RE-Bench.
  • Trend estimates: The full 2019–2025 trend had a measured doubling time of 207 days, compared with 172 days for non-SWAA tasks and 2023–2025 models.The difference remained within the margin of error.
  • Messiness: The evaluated HCAST and RE-Bench tasks averaged 3.2/16 on messiness, while a task such as writing a good research paper would score 9/16–15/16.The paper uses this contrast to contextualize the realism of the task suite.
  • Messiness: AI agents perform worse on messier HCAST tasks than task length alone predicts: each one-point increase in messiness reduced mean success rates by roughly 8.1%.The reported relationship was b=-0.081 with R2 = 0.251.
  • Messiness: Despite lower performance on messier tasks, success rates on high- and low-messiness subgroups both increased by 40 percentage points between January 2023 and May 2025.The comparison concerns sub-hour tasks and provides no evidence of a messiness-specific plateau or much slower trend.

H More ablations and robustness checks

Robustness checks largely preserve the reported horizon trends, while highlighting uncertainty from task measurement, extrapolation, and benchmark scope. Alternative analyses support exponential growth, but longer-task difficulty ratings and messiness measures remain important limitations.

  • Robustness checks: Methodological alterations leave the main results largely robust, despite some ablations using different baselines and task-success filtering.The ablations include alternative curve fits, task-distribution normalization, difficulty estimates, baseliner-ability checks, family weightings, training-date estimates, and continuous scoring.
  • Task messiness: Models perform worse on higher-messiness tasks, although improvement rates are similar across lower- and higher-messiness subsets.Messiness is negatively related to excess success rates, and the measure may be adversarially selected against current models.
  • Curve fitting: R2 ≥0.96 supports the exponential fit for horizon growth, whereas linear and hyperbolic fits perform poorly.The analysis uses an exponential curve because only 12 frontier models were measured and more expressive fits could overfit.
  • Continuous scoring: Nearly 2 hours is Claude 3.7 Sonnet’s continuous-scoring 50% horizon, but this approach likely overstates recent-model horizons and the growth slope.Continuous scoring captures more signal from 8-hour RE-Bench tasks, while longer tasks tend to be continuously scored.
  • Success-rate thresholds: 80% horizons have a similar doubling time to 50% horizons but are substantially lower.This provides a stricter reliability view of the same time-horizon trend.
  • Benchmark comparison: SWE-bench time annotations may underestimate human completion times, while less capable models show substantially longer horizons on these tasks than on SWE-bench Verified.The geometric mean of four sampled tasks in the “< 15 minute fix” bucket was 32.9 minutes.

H.7.2 Results

Messiness can affect humans and agents differently: in one task variant, o1 performed better when files were scrambled, despite a longer human baseline time. Cross-model task-success correlations remain positive after adjusting for expected success.

  • Task messiness: Messier-task success patterns are not uniformly harmful: o1 performed better on the messier report variant while human baseliners performed much worse.This single example illustrates why factors that hinder humans may sometimes improve agent performance.
  • Results: 50% success followed scrambling files, versus 34% for the local research report task, while baseliner time increased from 24 to 53 minutes.The scrambled-files variant had both higher agent success and a longer human completion time.
  • Results: o1 often succeeds on the scrambled variant by searching more generally after failing to find the expected “AR report” filename.When prompted with the expected term, it may grep for “AR report,” miss the actual “ARA report,” and guess.
  • Cross-model correlations: 0.38 versus 0.71: excess-success correlations are lower than raw task-success correlations but remain positive across models.The positive adjusted correlation suggests additional task factors common across models explain some success-rate variation.
Loading 2503.14499v4…