PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentTime: Can Agents Estimate and Control Their Own Runtime?

Michael Ofengenden, Maksym Andriushchenko

arXiv:2610.09944v2cs.AI

TL;DR

Agents need to estimate runtime, track elapsed time, and follow requested work durations, yet prior evaluations largely measure task duration rather than agents’ own runtime control. AgentTime tests these abilities across 222 tasks and finds that runtime-following varies across models, while task completion and time management remain distinct.

  • Problem

    Prior evaluations use human completion time as a proxy for task difficulty rather than measuring agents’ own runtime awareness or control, despite its relevance to following user instructions.

    Source: §1 · p. 2

  • Method

    AgentTime tests runtime forecasting, retrospective elapsed-time estimation, and duration-following across 222 tasks from 18 benchmark families in agents’ native harnesses.

    Source: §1 · p. 2

  • Results

    Astra ends 63% of runs on time, Sol 39%, and Fable 5.1 4%; forecasts usually exceed natural runtimes, and removing time cues increases retrospective errors.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Task completion does not establish control over time spent, so long-horizon evaluation should assess duration-following alongside task success and candor about how time was used.

    Source: §7 · p. 10

    Each task was generally run once per requested duration, leaving run-to-run variation unmeasured; only three agents from two labs were tested.

    Source: §6 · p. 9

Abstract

from arXiv · show

An essential control of AI agents is their ability to manage runtime. This ability requires a sense of time-awareness, to predict and estimate wall-clock time and to control their own actions. Prior work has focused on time-awareness, but duration-following and control in native agent harnesses remain unexplored. We present AgentTime, a benchmark for testing whether agents can work for a requested duration, predict their runtime, and estimate elapsed time afterward. It comprises 222 tasks from 18 sources spanning coding, computer use, agentic work, and automated research. Duration-following experiments append a single instruction specifying how long to work, with requests ranging from about a minute to multiple days. Accuracy on these instructions varies substantially: Fable 5.1 in Claude Code deviates from requested runtimes by a typical factor of 2.9$\times$, compared with only 1.2$\times$ for GPT-6 Astra in Codex. However, matching the requested runtime does not, by itself, establish continued work on the task. Among 158 reviewed Astra runs with classifiable transcripts, 14 explicitly slept after appearing to finish. In forecasting experiments, predictions tend to overestimate natural runtimes. In retrospective experiments, removing temporal information more than doubles deviation for Sol and Astra and nearly doubles it for Fable. An agent's ability to complete a task does not guarantee that it can control its own time or work for the whole requested duration. For agents to run reliably, safely, and autonomously over long horizons, we require the evaluation of both.

1 INTRODUCTION

AgentTime treats runtime control as distinct from task completion: agents must estimate time, follow requested durations, and account for elapsed time. Earlier evaluations use human completion time as a proxy for difficulty rather than measuring agents’ control or awareness of their own runtime.

1 INTRODUCTION

  • Reliable, user-responsive agents need temporal self-awareness, but control over execution time is dual-use and can itself become an object of agent investigation.The passage cites reports of agents probing run endings with an external heartbeat and sharing information across runs.Source: §1 · p. 2
  • Prior evaluations use human task-completion time as a proxy for difficulty but do not measure agents’ control or awareness of their own runtime.Because an LLM has no internal clock, it must infer time from context and may spend substantial compute without registering elapsed time.Source: §1 · p. 2
  • Forecasting, retrospection, and duration-following are distinct skills, and agents need all three to budget effort across a requested interval.Accurate estimates alone do not ensure that an agent uses the remaining time rather than stopping or sleeping.Source: §1 · p. 2
  • AgentTime evaluates runtime forecasting, elapsed-time estimation, and duration-following across 222 tasks from 18 benchmark families spanning coding, computer use, agentic work, and automated research.Duration requests range from about a minute to 60 hours.Source: §1 · p. 2
  • GPT-6 Astra in Codex follows requested runtimes closely, but 14 of 158 reviewed runs slept after appearing finished; forecasts tend to run high, and removing time cues worsens retrospective estimates.Without time cues, typical retrospective misses grow fivefold for Sol, 2.5-fold for Astra, and 1.9-fold for Fable.Source: §1 · p. 2

2 RELATED WORK

Related evaluations examine temporal reasoning, resource budgets, and situational awareness from different angles. AgentTime focuses on an agent’s own execution time and whether it works for a requested interval.

2 RELATED WORK

  • Prior runtime-estimation studies report substantial errors across models and tasks (Garikaparthi, 2026); AgentTime tests runtime estimates and duration-following in native CLI harnesses, including long-running tasks.The paper evaluates agents with their available tools rather than treating temporal reports as evidence of an unaided internal clock.Source: §2 · p. 3
  • Budget and deadline evaluations study constrained planning and tool use, while AgentTime distinguishes completing within an interval from working for its full duration and records sleeping separately.The related evaluations include explicit tool-call budgets, resource estimates, deadline negotiation, and wall-clock budgets.Source: §2 · p. 3
  • AgentTime asks agents how long their own tasks took and whether they worked for the full requested period, unlike evaluations of event timing or conversational memory.Those other evaluations ask when events happened or whether information remains relevant; an agent can remember events yet misjudge elapsed time.Source: §2 · p. 3
  • Situational-awareness and reasoning-control work motivates examining an agent’s temporal state, while AgentTime measures response to requested wall-clock work duration in native harnesses.The paper does not evaluate duration-following as a means for deception, but does measure the underlying capability.Source: §2 · p. 3
  • Table 1 marks which capabilities evaluations test, not whether agents succeed; its distinctions include time or budget measurement, budget action, and multi-hour runs.The table caption defines “multi-hour” as individual measured wall-clock runs lasting multiple hours.Source: Table 1 · p. 4

3 BENCHMARK AND SETUP

AgentTime combines tasks from existing benchmarks with native scoring and external runtime measurement. Its setup includes duration-following runs, preliminary ablations, and fresh agent sessions in benchmark-matched environments.

3 BENCHMARK AND SETUP

  • The suite contains 222 tasks from 18 benchmarks, with three requested-duration runs per task and durations chosen to include too-little-time and ample-time extremes.Durations typically increase in steps of four; the shortest and longest requests vary across benchmarks.Source: §3 · p. 4, Table 2 · p. 4
  • Benchmark selection spans question answering, software engineering, computer use, web research, document editing, and automated research, enabling native task scores alongside duration adherence.The study retains each benchmark’s native grading rules and adds no pass mark for the seven benchmarks without one.Source: §3 · pp. 4–5
  • Preliminary ablations use ProgramBench’s 200 difficult re-implementation tasks, which provide long runtimes for studying temporal self-control.Tasks include a compiled binary and documentation, and prohibit internet access.Source: §3 · p. 5
  • Runs begin in fresh Docker containers with benchmark-matched files, tools, and network access; external timers measure runtime and graders score results after sessions end.Native time caps and budgets are removed, and duration instructions are appended to prompts.Source: §3 · p. 5

4 EXPERIMENTS AND RESULTS

Duration-following, natural-runtime forecasting, and elapsed-time retrospection are evaluated separately. Results show that duration adherence varies, forecasts generally overestimate runtime, and removing temporal cues worsens retrospective estimates.

4.1 DURATION-FOLLOWING: EARLY RETURN AND EXPLICIT WAITING

  • 63% of Astra runs, 39% of Sol runs, and 4% of Fable runs ended within 5% of the requested duration.Each run used a fresh session and workspace; the request was an instruction, not a forced stopping time.Source: §4.1 · p. 5
  • Asked for 16 times more time, Fable ran 1.9 times longer, Sol 5.0 times, and Astra 12.1 times within task.On swapped-harness tasks, Astra’s within-task slope exceeded Fable’s by 0.71 in Claude Code and 0.69 in Codex.Source: §4.1 · p. 6
  • 62% of Fable, 63% of Sol, and 64% of Astra tasks had the same native score at the longest and shortest requests.Across 158 classifiable Astra transcripts, 14 runs explicitly slept after appearing to finish; a good score does not establish work throughout the requested period.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 FORECASTING NATURAL RUNTIMES

  • 83% of Sol forecasts, 66% of Astra forecasts, and 63% of Fable forecasts exceeded natural runtimes; typical forecast errors were 3.8×, 2.9×, and 2.6×, respectively.The forecasts showed little central tendency: Sol’s overestimate was about 3 times, compared with 1.3 for Fable and 1.4 for Astra.Source: §4.2 · pp. 7–8

4.3 ESTIMATING TIME SPENT

  • Removing timestamps and other time cues increased Sol’s typical miss from 1.07× to 5.38×, while Fable’s and Astra’s reached about 2.6×.Examined transcripts contained cues such as tool-call durations and timestamps; in a preliminary Claude Code study, visible event count correlated with runtime at r = 0.91.Source: §4.3 · pp. 8–9

5 DISCUSSION

Runtime control varies largely by model, with Astra outperforming the other tested models and Fable’s harness swaps preserving the model-linked difference. Agents still struggle to predict task-work duration, while explanations for their timing behavior and preferred runtime use remain unresolved or task-dependent.

5 DISCUSSION

  • Runtime control varies largely by model: Astra makes significant progress over the other tested models, and Fable 5.1’s harness swaps preserve the model-linked difference.The experiments do not distinguish whether timestamped tool calls or assumptions about generation speed shape runtime estimates.Source: §5 · p. 9

5.1 WHERE WOULD TIME-AWARENESS AND CONTROL OF RUNTIME COME FROM?

  • Agents still struggle to predict task-work duration, and the experiments do not distinguish timestamped-tool-call learning from generation-speed assumptions as possible explanations.Source: §5.1 · p. 9, §5 · p. 9
  • Preferred runtime behavior depends on the task: fast answers may suit closed-book questions, while open-ended work may benefit from using the requested time.The paper hypothesizes that emphasizing duration instructions in reinforcement-learning environments could improve execution-time control.Source: §5.1 · p. 9

6 LIMITATIONS

The study’s results are bounded by limited replication and model coverage, and very long requests and harness swaps cover only a small portion of the evaluation. The campaign also required substantial compute and cost.

6 LIMITATIONS

  • Each task-duration condition was run once, so results mix responses to duration requests with run-to-run variation; only 55 question runs were repeated.The study tested three agents from two labs, with one duration-instruction wording at maximum reasoning effort.Source: §6 · p. 9
  • Requests beyond 8.3 hours occur only in METR and PostTrainBench, covering four tasks total; harness swaps cover requests up to 80 minutes.Source: §6 · p. 9
  • The duration-following experiment used about 1,700 agent-hours, with estimated total campaign costs of about $93,000 at list prices.The estimate includes about $86,000 in model access and about $7,000 for rented hardware.Source: §6 · p. 9

7 CONCLUSION & FUTURE WORK

AgentTime finds substantial variation in runtime control, forecasting, and retrospective estimates. The results distinguish task completion from time management and motivate evaluating duration-following and candor, while leaving safety-related questions for future work.

7 CONCLUSION & FUTURE WORK

  • GPT-6 Astra ended 63% of runs on time, GPT-5.6 Sol 39%, and Fable 5.1 4%; forecasts were mostly too high, while removing time cues increased typical retrospective misses about two- to fivefold.Some reviewed Astra runs appeared to finish and then sleep; a request about sixteen times longer raised native scores on only about a quarter of tasks.Source: §7 · p. 9
  • Task completion and time management are distinct: overrunning, stopping early without saying so, or quietly waiting can make agents harder to direct over long horizons despite good answers.Source: §7 · p. 10
  • Duration-following should be evaluated alongside task success for long-horizon agents, together with candor about how time was spent.Source: §7 · p. 10
  • Whether these behaviors predict conduct under oversight remains open; early returns and explicit sleeping are not evidence of monitor evasion.Future work could test whether agents report finishing early or waiting, and whether generation-speed knowledge or faster inference improves estimates.Source: §7 · p. 10

A NATIVE SCORING AND SUCCESS DEFINITIONS

The scoring appendix documents how the evaluation preserves benchmark-native grading and handles score scales, incomplete grades, and several benchmark-specific exceptions. It also details additional grading procedures and constraints for selected benchmarks.

A NATIVE SCORING AND SUCCESS DEFINITIONS

  • The scoring map retains each benchmark’s own pass/fail rule where available and adds none for benchmarks without one; scores are mapped to percentages when their range is fixed.Eleven of eighteen benchmarks define a pass/fail rule; ALE-Bench and YC-Bench are compared only within tasks.Source: Table 4 · p. 15, A NATIVE SCORING AND SUCCESS DEFINITIONS · p. 15
  • Table 5 pools mean native scores across requested durations; its suite comparison uses 604 graded cells from 16 benchmarks, while some benchmark scores rest on few runs.Of 1,991 runs, 1,911 had native scores and 56 additional runs scored zero for leaving nothing to grade.Source: A NATIVE SCORING AND SUCCESS DEFINITIONS · pp. 15–16, Table 5 · p. 16
  • Some grades were collected after the campaign using benchmark-native graders or score definitions applied to the outputs runs left behind.For some runs, outputs were read from disk snapshots or saved collection records.Source: A NATIVE SCORING AND SUCCESS DEFINITIONS · pp. 16–17
  • OSWorld 2.0 task 092 scores at most 0.8 because its video check was skipped for all nine runs; AssistantBench scores answers from submitted files or transcript calls.Eleven Sol runs without an answer tool were scored from the final answer they saved; runs without a submitted answer score zero.Source: A NATIVE SCORING AND SUCCESS DEFINITIONS · p. 17
  • For Agents’ Last Exam, eight sealed outputs received the same scores under the unchanged evaluator; ProgramBench’s first-pass date issue caused 14 of 47 submissions to fail building before regrading.The ProgramBench regrade dated files to 1 January 2000; two runs with both sealed and machine-left files had matching scores.Source: A NATIVE SCORING AND SUCCESS DEFINITIONS · p. 17
  • PaperBench’s 19 graded runs used GPT-6 Luna instead of campaign judge o3; on two shared submissions, scores differed by −5.6 and +0.7 points, with 97% rubric-leaf agreement.Runs received four hours on an H100, and six of 19 reproductions reached that limit.Source: A NATIVE SCORING AND SUCCESS DEFINITIONS · pp. 16–17

B SWAPPING HARNESSES UNDER DURATION REQUESTS

Harness swaps show that model identity accounts for more variation in duration-following than harness choice in the tested runs. Preliminary forecasting studies found compressed predictions, while separate-turn prompts and retrospective transcript conditions provide additional study designs.

B SWAPPING HARNESSES UNDER DURATION REQUESTS

  • In Codex, Fable often spent longer and produced more tokens than in Claude Code, without improving its question-run accuracy.Its median output was 1.6 times larger in Codex; it answered 43 of 54 question runs correctly there and 48 in Claude Code.Source: B SWAPPING HARNESSES UNDER DURATION… · p. 18
  • Astra ended fewer runs on time in Claude Code than Codex, and all 72 misses in Claude Code ran late; OpenRouter may explain part of the difference.Claude Code on-time counts were 12 of 54 question runs and 14 of 44 agentic runs, compared with 35 of 54 and 25 of 44 in Codex.Source: B SWAPPING HARNESSES UNDER DURATION… · p. 18, Table 6 · p. 20
  • Fable was on time in 1 of 98 Codex runs and 5 of 98 Claude Code runs, versus Astra’s 60 of 98 Codex runs; swapping harnesses changed within-task slopes less than model identity did.Fable’s slope gap from Astra was about 0.7, compared with a harness effect of about 0.15 on these 34 tasks.Source: B SWAPPING HARNESSES UNDER DURATION… · p. 18, Figure 5 · p. 19, Figure 6 · p. 19

C MEASURING NATURAL RUNTIME

  • Natural-runtime forecasts were elicited in separate sessions from the 222 no-duration task runs, with all Astra and Sol prompts and 216 Fable prompts returning numbers.Forecasts were requested at maximum reasoning effort in each agent’s native harness.Source: C MEASURING NATURAL RUNTIME · p. 23

D RETROSPECTIVE CONDITIONS

  • Retrospective conditions vary access to timing evidence through session forks, reconstructed transcripts, and a scrubbed transcript without timestamps.Replay and Scrubbed use reconstructed logs sent to the API without tools; All disabled turns off every tool.Source: D RETROSPECTIVE CONDITIONS · p. 23

E PRELIMINARY STUDIES

  • The preliminary studies used earlier models and mostly ProgramBench, so their findings are reported separately rather than pooled with the main experiments.They informed the design of the main experiments.Source: E PRELIMINARY STUDIES · p. 23

E.1 SAME-TURN FORECASTS

  • Opus 4.8 averaged 85 minutes on ProgramBench, while GPT-5.5 averaged 17.5 minutes; these are measured model–harness systems, not isolated harness effects.The study used a same-turn estimate prompt before task completion.Source: §E.1 · p. 23

E.1.1 OPUS 4.8 & GPT-5.5 LIKE TO PREDICT AROUND “AN HOUR AND A HALF”

  • Both models’ forecasts varied little with task duration, and their low fitted exponents indicate compressed variation rather than accurate task-specific estimates.A slope of one alone does not establish accuracy; proportional scaling also requires zero intercept and residual error.Source: §E.1.1 · p. 24, Figure 7 · p. 24

E.2.1 GPT-5.6 SOL’S RUNTIMES PLATEAU NEAR 30 MINUTES

On ProgramBench, Sol tends to stop after about thirty minutes regardless of task complexity, whereas Opus 5 runs longer and has higher first-turn accuracy. Forecasts are generally too long, and additional forecast evidence does not consistently improve calibration.

E.2.1 GPT-5.6 SOL’S RUNTIMES PLATEAU NEAR 30 MINUTES

  • Sol stops after about thirty minutes with little regard to task complexity, while Opus 5 has a median runtime of 93 minutes and higher first-turn accuracy.Opus 5’s first-turn accuracy was 96.8%, versus Sol’s 82.0%.Source: §E.2.1 · p. 25
  • Both agents tend to forecast longer runtimes than they actually use, while the sample comparisons show Sol’s stopping tendency against Opus’s more linear runtime pattern.The figure compares forecast and measured runtime for a sample of tasks.Source: §E.2.1 · p. 25, Figure 8 · p. 25
  • Giving either model more evidence does not consistently reduce forecast error; for Sol, more evidence systematically lengthens predictions.The information conditions vary what is available before the task.Source: §E.2.1 · p. 25, Figure 9 · p. 26

E.3 WHOSE PRIOR IS IT?

Forecast wording experiments find that the estimated referent matters more than minor phrasing changes, with human estimates often much longer than self-estimates. Transcript logs also contain multiple kinds of timing evidence that agents can inspect.

E.3 WHOSE PRIOR IS IT?

  • Changing the forecast referent shifts estimates more than changing equivalent wording, while estimates for skilled humans are about four times Fable’s self-estimates and three times Sol’s.Swapping “complete” and “finish” shifted estimates by about 10%; changing “skilled human professional” to “human expert” had much smaller effects.Source: Figure 10 · p. 26, §E.3 · p. 26
  • The perceived agent advantage grows with task scale, reaching roughly tenfold on longer tasks, except on OSWorld where both systems recognize human experts as superior.On short tasks, neither Claude Code nor Codex claims a time advantage.Source: §E.3 · p. 26, Figure 11 · p. 27
  • Models give different kinds of reasons depending on who is completing the task, and forecasts for longer tasks are closer to agents’ actual completion times.The figure describes Fable’s explanations by referent and categories that may overlap.Source: Figure 12 · p. 27, §E.3 · p. 27

E.4 TIMESTAMPS

  • Every reviewed log contained timing evidence, including tool-call durations, absolute timestamps, or time-related reasoning; many agents searched transcripts for clock readings.The listed time cues include finished-tool-call durations and dated files or logs.Source: §E.4 · p. 28

E.5 RANKING TASK-LENGTH VERSUS A SIMPLE BASELINE

A preliminary comparison finds that transcript event counts correlate with true runtime more strongly than after-run reports in both harnesses, but the correlations do not establish that reports add information. The associated self-rating procedure asks for an effort-based score without access to official task evaluation.

E.5 RANKING TASK-LENGTH VERSUS A SIMPLE BASELINE

  • These correlations do not establish whether after-run reports add information beyond transcript length.Source: §E.5 · p. 28
  • Visible event count correlated with runtime at r = 0.91 in Claude Code and r = 0.52 in Codex, compared with after-run reports at r = 0.86 and r = 0.48, respectively.The preliminary study used 200 runs per harness.Source: Figure 14 · p. 28, §E.5 · p. 28

E.6 DO AGENTS KNOW HOW WELL THEY DID?

  • The self-rating procedure asks agents to report a score estimated from their own sense of effort, and the separate-turn follow-up has no access to the test suite or final score.Source: §E.6 · p. 29

E.6.1 OPUS 4.8 AND GPT-5.5 ARE OVERCONFIDENT IN THEMSELVES

Opus 4.8 and GPT-5.5 often overrate their own benchmark performance, while Sol and Opus 5 show contrasting self-assessment errors. The paper also reports extensive AI assistance in its research process, alongside human review of the final work.

E.6.1 OPUS 4.8 AND GPT-5.5 ARE OVERCONFIDENT IN THEMSELVES

  • 20 percentage points: Opus 4.8 and GPT-5.5 overrate their final scores on average, with little task-level correlation and high estimates even after failures.Both often predict scores near 80% when official scores are lower.Source: §E.6.1 · p. 29, Figure 15 · p. 29
  • Sol consistently overestimates its performance, while Opus 5 typically underestimates by more than 10 percentage points.Sol gave scores in the mid-90s for some runs below 50%; Opus 5 estimated 86% on runs scoring 99.5% and 96.7%.Source: §E.6.1 · p. 30, Figure 16 · p. 30
  • On a php-src case, both models guessed around 70% despite official scores of 7% and 14.5%.Source: §E.6.1 · p. 30

F AI USE STATEMENT

  • Generative AI tools supported implementation, data work, analysis, coding, figure creation, and editing, and the authors reviewed all AI-assisted work.A separate Claude model recomputed Table 3 from raw run records, matching all values except one rounding error that was fixed.Source: F AI USE STATEMENT · p. 30
Loading 2610.09944v2…