PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Clock: Towards Punctual and Productive Time-Budgeted AI Agents

Aaron Wang, Neelabh Madan, Vlad Sobal, Matthew Trager, Michael Kleinman, Elman Mansimov, Wei Xia, Stefano Soatto

arXiv:2610.10833v1cs.AIcs.LG

TL;DR

The paper asks whether agents can obey explicit wall-clock budgets and use additional time productively. It compares harness mechanisms that expose or enforce timing with budget-aware GRPO training, finding that adherence improves substantially while productive allocation remains difficult.

  • Problem

    It is unclear whether LLM agents can both meet explicit wall-clock budgets and convert additional time into better solutions.

    Source: §1 · pp. 1–2

  • Method

    The paper compares harness mechanisms that expose or enforce timing with GRPO training that optimizes budget-conditioned behavior.

    Source: §6 · p. 8

  • Results

    Harness timing information substantially improves adherence, and GRPO matches Zork budgets within 0.2 s across 5–90 s budgets, including four held-out budgets, but extra time often does not improve task outcomes.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Adherence and productive time allocation are distinct: agents can learn when to stop without learning how to use additional time productively.

    Source: §6 · p. 9

    The evaluation is limited in models and tasks, and fixed training environments and strategy collapse across budgets constrain the reported findings.

    Source: §5 · p. 8, §6 · p. 9

Abstract

from arXiv · show

We study whether small LLM agents can operate effectively under explicit wall-clock time budgets by both respecting the allocated runtime and using available time productively. We evaluate Qwen3.6-27B on five competitions from MLE-Bench Lite and Qwen3-4B on Zork I (Jericho), two agentic benchmarks where additional computational time can meaningfully improve performance. In the simplest setting, where the budget is stated only in the prompt, agents fail to translate the stated budget into controlled use of time. These failures arise from gaps in time awareness, since the harness provides no timing feedback, but also because they cannot reliably anticipate the duration of actions, and do not have a learned mapping from available time to an appropriate strategy. We investigate two complementary classes of interventions: harness-based mechanisms that expose timing information and enforce deadlines, and reinforcement learning with budget-aware rewards. Injecting timing information through the harness substantially improves budget adherence for Qwen3.6-27B without measurable loss in performance, while enforcement hooks tighten adherence further. RL with GRPO achieves near-perfect budget adherence on Zork I and generalizes to held-out budgets not seen during training, but does not improve task performance over the untrained harness on MLE-Bench. Once agents are made to respect the budget, they still fail to use additional time to improve task performance. RL-trained policies learn when to stop but often fill extra time with repeated actions, and GRPO training on multiple budgets tends to collapse toward the strategy learned for the shortest budget. Our results reveal a gap between time adherence and productive time allocation, which remains a central challenge for budget-conditioned agents.

1 Introduction

The paper asks whether agents can obey explicit wall-clock budgets while converting available time into better solutions. It finds adherence more tractable than productive use, motivating separate harness and learning interventions.

1 Introduction

  • Budgeted agents need both adherence—returning within the allocated time—and productive use, spending extra time to improve solutions rather than stopping early or padding runtime.A concrete test of productive use is whether task performance improves as the budget increases.Source: §1 · p. 2
  • With only a prompt-stated budget, Qwen3.6-27B runs for roughly ten minutes regardless of the limit, overrunning an 8-minute budget while using little of a 60-minute budget.The Dog Breed Identification results report runs of 8–13 minutes regardless of the prompt.Source: §1 · p. 2, Table 1 · p. 2
  • Agents lack elapsed-time feedback, misestimate their own elapsed time and action durations, and may still need to learn how available time should shape a plan.Even with a clock in context, forecasts of remaining and total run duration stay far off.Source: §1 · p. 2
  • The paper tests harness interventions that expose and enforce timing and GRPO training with budget-aware rewards across five MLE-Bench Lite competitions and Zork I.The selected tasks are intended to reveal how different budgets may favor different strategies.Source: §1 · pp. 2–3
  • The harness brings Qwen3.6-27B from 161% of budget to on-budget with no measurable loss in quality, while RL achieves near-exact Zork I adherence but does not reliably turn extra time into better outcomes.RL-trained policies can fill additional time with actions that do not improve task results.Source: §1 · p. 3

2 Related Work

Related work spans budget-aware reasoning and search, wall-clock test-time scaling, constrained reinforcement learning, and RL methods for language models. This paper distinguishes itself by studying wall-clock time and comparing prompting, harness enforcement, and RL.

2 Related Work

  • Prior budget-awareness and search work studies token or tool-call budgets, whereas this paper studies stochastic wall-clock durations that token counting cannot capture [1] [2] [3].The cited work includes progressive interval estimation, Budget-Aware Value Tree Search, and analyses of search depth and completion budgets.Source: §2 · p. 3
  • Unlike work on compute-optimal scaling with unlimited budgets or a single time-aware RL intervention, this paper compares prompting, harness enforcement, and RL [4] [5] [6].It also studies how strategy should adapt to a time budget, finding that RL learns when to stop but not what to do differently with more time.Source: §2 · p. 3
  • Budgeted MDPs condition policies on budget, while anytime constraints motivate exposing remaining budget; prior work also warns that reward-driven agents may brute-force without transferable strategies [8] [9] [10].The paper’s RL-trained policy exhibits a memorized prefix followed by no-ops.Source: §2 · p. 3
  • The paper adopts GRPO, which normalizes rewards across grouped rollouts without a separate critic, to handle multimodal rewards across budget levels [11] [12].Source: §2 · p. 3

3 Setup

The setup evaluates time-budgeted agents in machine-learning engineering and interactive fiction, where additional wall-clock time can support different strategies and improve scores. It varies available time and measures trajectories and final performance.

3 Setup

  • The evaluation uses five selected MLE-Bench Lite competitions, where short budgets may favor baseline models and longer budgets allow analysis, feature engineering, model selection, tuning, and longer training.The selected tasks span image, leaf-feature, text, and tabular prediction.Source: §3.1 · p. 4
  • MLE-Bench tasks require inspecting data, developing and running modeling code, iterating on experiments, and producing valid submissions; Qwen3.6-27B is run through the OpenCode harness.The study varies wall-clock budgets and measures trajectory and final task performance.Source: §3.1 · p. 4
  • In Jericho’s Zork I, the agent takes natural-language actions and receives game-state observations; more time permits additional exploration, map discovery, and puzzle solving.Source: §3.1 · p. 4
  • Zork evaluations use a lightweight agentic loop and normalized game score, with a maximum of 350 points; RL trains policies to trade off exploration, progress, and budget adherence.Evaluated models include Qwen3-4B, Qwen3.5-9B, and Qwen3.8-27B.Source: §3.1 · p. 4

4 Results

Across MLE-Bench and Zork I, prompt-only agents do not reliably stay within wall-clock budgets, while harness feedback improves adherence. Even after training or enforcement helps agents stop on time, using additional time to improve task performance remains difficult.

4.1 Prompting

  • At a fixed 15-minute budget across five MLE-Bench competitions, prompt-only agents used 161% of the budget on average and only 15% of runs finished on time.The table reports on-budget rate, quality among valid on-budget submissions, turns, and time.Source: §4.1 · p. 4, Table 2 · p. 5
  • At 16 s, the trained Zork policies scored 46.4 and 16.6 points, while no fixed-weight condition exceeded 0.6 points at any budget.The figure compares mean scores and elapsed time for Qwen3-4B; RL achieved the strongest overall performance and budget adherence.Source: Figure 2 · p. 5

4.2 Interventions via Harness

  • Harness timing feedback reduced MLE-Bench time use from 161.4% to 99.7% of budget without measurable quality change, while valid submissions rose from 78% to 88%.Shortening command limits preserved most unenforced quality and yielded the highest measured valid-submission rate, 90%; outright blocking reduced that rate.Source: §4.2 · pp. 5–6
  • Even without RL, elapsed time scaled to the stated budget when the harness supplied a clock, but quality showed no upward trend as budgets grew.AutoClock provided timing after every turn; the harness could not intervene during a long generation, and scores remained below 5 across budgets.Source: §4.2 · p. 6

4.3.1 MLE-Bench

  • On MLE-Bench, GRPO training did not improve over its harness baseline, though interaction steps fell from 20.3 to 18.9 and budget use stayed between 0.87 and 0.93.About a quarter of on-time submissions scored no better than a uniform guess, and that fraction did not fall when the quality weight was tripled.Source: §4.3.1 · p. 7

4.3.2 Zork I

  • On Zork I, the selected step-wise reward trained policies against budgets sampled uniformly from 0 to 16 s; Clock Tool++ scored below AutoClock while spending roughly 50% of turns checking time.Both trained policies typically terminated within ±15% of the specified budget.Source: §4.3.2 · p. 7
  • Mixed-budget training produced nearly perfect adherence across horizons, but game scores followed nearly identical trends as budgets grew.The policy learned when to stop for each budget, not how to use additional time productively.Source: §4.3.2 · pp. 7–8, Figure 3 · p. 8

5 Limitations

The limitations center on policies that adapt stopping time without adapting strategy, memorization in a fixed Zork setting, and a narrow evaluation scope.

5 Limitations

  • Across budgets, the trained Zork policy follows the same action sequence and fills extra time with filler rather than planning differently.Mixed-budget training helps it generalize stopping across budgets, but not strategy.Source: §5 · p. 8
  • Training on one game with a fixed starting state favors a recalled Zork walkthrough, and the policy scores far lower when valid actions are fed back.The elapsed-time reward does not price exploration or distinguish rejected from productive actions, so the policy can repeat commands.Source: §5 · p. 8
  • The evaluation covers two models and two environments, with Zork RL trained on one game and MLE-Bench limited to five competitions.Whether findings extend to frontier-scale models or richer strategy spaces remains open.Source: §5 · p. 8

6 Conclusion

The paper compares harness-based timing mechanisms with GRPO training for agents given wall-clock budgets. Harness clocks improve adherence, while RL tightens it further; using added time productively remains unresolved.

6 Conclusion

  • The study compares harness mechanisms that expose or enforce timing with GRPO training, after finding that prompt-only budgets do not ensure agents estimate action durations reliably.The evaluation uses five MLE-Bench Lite competitions and Zork I (Jericho).Source: §6 · p. 8
  • Informational clocks reduce overruns to near zero in both environments with little measurable loss in task quality, and stricter enforcement tightens adherence at a modest performance cost.Prompt-only agents overshoot small budgets by up to 2× on Zork and 61% on MLE-Bench.Source: §6 · p. 9
  • Within 0.2 s of requested budgets from 5 to 90 s, GRPO-trained Zork policies generalize to four held-out budgets, while larger-budget scores saturate at 45/350.On MLE-Bench, trained policies use 0.87–0.93 of the allocated budget; extra time yields more attempts at essentially the same training procedure without improving submission quality.Source: §6 · p. 9
  • The results distinguish budget adherence from productive allocation: agents can stop on time without changing behavior to improve solutions when given more time.The authors identify strategy collapse, memorization in fixed environments, and limited evaluation scope as limitations.Source: §6 · p. 9

Reward structure and exploration incentives

The reward structure values final score and elapsed time, but does not directly reward exploration or distinguish repeated attempts from different strategies.

Reward structure and exploration incentives

  • In Zork, the reward prices final score and elapsed time but leaves exploration unpriced, making deviation from a scoring prefix risky.With λ = 0.1, perfect adherence is worth about seven times a genuine five-point discovery.Source: Reward structure and exploration… · p. 11
  • A five-point Zork gain increases the score term by only 5/350 ≈0.014, while the time term can be worth up to 0.100.The policy typically reaches at most about 45/350 ≈0.129 in score.Source: Reward structure and exploration… · p. 11
  • On MLE-Bench, repeating the same baseline three times can receive the same reward as trying progressively more sophisticated methods when the best submission score is identical.The reward does not distinguish repeated attempts from genuinely different approaches.Source: Reward structure and exploration… · p. 11

Budget adherence through invalid actions

The Zork policy meets its wall-clock target largely by repeating rejected commands, so clock use can look compliant without representing useful work.

Budget adherence through invalid actions

  • Rejected “forward” appears in 127 of 128 rollouts, accounting for 45% of turns at B = 15 and 88% at B = 90.It begins immediately after the memorized prefix ends.Source: Budget adherence through invalid actions · p. 11
  • The policy meets the budget to within 0.2 s almost entirely through repeated actions that advance nothing.Rejected commands consume the same generation time as valid moves.Source: Budget adherence through invalid actions · p. 11
  • Wall-clock utilization is not a measure of effort, and the tail-only repetition detector misses hundreds of consecutive no-ops.The authors suggest measuring state-changing turns, valid-command rate, or tokens spent on accepted actions.Source: Budget adherence through invalid actions · p. 11

Comparing Reward Shapes

The linear, Gaussian, and step-wise time-reward variants perform comparably overall, but the step reward matches budgets most reliably.

Comparing Reward Shapes

Action durations on MLE-Bench

On MLE-Bench, stochastic action durations create reward noise unrelated to agent decisions, while the harness-only condition leaves little room for RL gains.

Action durations on MLE-Bench

  • The harness-only condition achieves 99.7% budget utilization, leaving little room for RL to improve.RL training on MLE-Bench produced negligible gains over the untrained harness.Source: Action durations on MLE-Bench · p. 11
  • The same MLE-Bench training script may take 30 seconds or 3 minutes depending on system load.Action durations vary with factors outside the agent’s control, including GPU availability, data loading, and network latency.Source: Action durations on MLE-Bench · p. 11
  • Unpredictable action durations make the RL reward noisy in ways uncorrelated with the agent’s decisions, complicating credit assignment [9].Source: Action durations on MLE-Bench · p. 11

Environment and Interaction Protocol

This section describes the agent interaction and evaluation setup, including time budgets, clock access, elapsed-time and task-quality measurement, and the MLE-Bench episode configuration. It also specifies budget-conditioned prompts and tool feedback, the grouped GRPO training protocol and rewards, and episode-reuse probes for time estimation.

Environment and Interaction Protocol

  • Each turn supplies the game response, score and move status, and a fixed question; the agent returns REASONING and ACTION lines, with parsing fallbacks and a 190-character command limit.Episodes end on agent quit, game over or victory, 4,000 turns, wall-clock cutoff, context overflow, or LLM error; the end reason distinguishes voluntary quit from harness cutoff.Source: Environment and Interaction Protocol · p. 12

Time Budget, Cutoff, and the Definition of Elapsed Time

  • The harness cutoff is 2B, checked before each generation, so episodes can overrun; budgets below about 2 seconds are used only to measure clock behavior, not score.Elapsed time includes generation latency, environment stepping, and overhead. One generation takes about 0.2–0.4 seconds for Qwen3-4B and 1.4–1.8 seconds for Qwen3.6-27B; overrun is reported relative to the cutoff.Source: Time Budget, Cutoff, and the… · p. 12

The Clock Interventions

  • Clock conditions differ in how the budget reaches the model, while sharing the same stated budget and 2B cutoff; clock checks consume generation time but neither advance the game nor use a move.The clock tool is a text-protocol pseudo-tool. Its reply matches the per-turn clock line, letting the comparison isolate push versus pull; the detailed prompt additionally advises early checks and adapting actions to the remaining budget.Source: The Clock Interventions · pp. 12–13, Table 4 · p. 13, Prompts · p. 15

Reinforcement Learning

  • GRPO uses normalized game score plus a λ = 0.1 step-wise time bonus, with the entire reward gated to zero when elapsed time exceeds 1.1B.The bonus saturates across [0.9B, 1.1B], making overshooting costly to both score and bonus, while undershooting loses only partial bonus credit; policy loss excludes environment-authored turns.Source: Reinforcement Learning · p. 13

Agent, Model Serving, and Sandbox

  • The MLE-Bench agent is headless OpenCode 1.17.4 using built-in tools and its own system prompt; it has no callable clock, and episodes run in isolated Docker containers.Qwen3.6-27B is served by SGLang with tensor parallelism 4; per-episode workspaces are fresh, public files are read-only, and processes are killed before grading.Source: Agent, Model Serving, and Sandbox · p. 17

Episode, Budget, and Elapsed Time

  • For MLE-Bench, elapsed time runs from workspace readiness until exit or kill and includes generation, tool execution, and harness overhead; an episode is one OpenCode invocation.The Table 2 kill is at 2B, while the later RL run uses B + 30 seconds; on-budget means t ≤ B, and valid measures the fraction of valid submissions.Source: Episode, Budget, and Elapsed Time · pp. 17–18

Grading and Quality

  • MLE-Bench quality is leaderboard percentile, while grading checks submission validity and returns each competition’s lower-is-better metric.The metric is multi-class log-loss for dog-breed, spooky-author, and leaf classification; log-loss for dogs-vs-cats; and mean column-wise RMSLE for NOMAD 2018.Source: Grading and Quality · pp. 18–19

Table 1 and Figure 1 (left)

  • The earlier dog-breed budget-in-prompt runs used a separate launcher without plugins, resume loop, or kill; each budget had its own prompt, while the no-budget control omitted time statements.Those episodes ended when the agent exited.Source: Table 1 and Figure 1 (left) · p. 19

Reinforcement Learning

  • The reward scales speed by quality and separates on-time, late, and invalid submissions, ensuring the worst on-time valid reward exceeds the best late valid reward on every competition.The late penalty decreases strictly with lateness and is bounded; the invalid penalty grows with time spent running without a file.Source: Reinforcement Learning · p. 20
  • GRPO groups episodes by competition and budget, yielding 20 groups of 21 episodes (420 episodes per optimizer step) across five competitions and four budgets.The four budgets are 10, 15, 20, and 25 minutes; Dogs-vs-cats is excluded from tables, and truncated importance sampling corrects generation-to-update staleness.Source: Reinforcement Learning · p. 20

Time-Estimation Probes

  • Time-estimation probes truncate recorded 15-minute episodes and ask models to estimate elapsed time, remaining budget, time until completion, and total duration.The clock condition keeps or removes harness clock lines; both conditions use the same episodes, and responses that continue tool calls are dropped.Source: Time-Estimation Probes · pp. 20–21

Prompts

  • Harness tool results appended generation time and remaining budget after every call, and displayed a budget-enforcement notice only when the clamp hook lowered a command’s time limit.The notice specified the reduced limit and reserved time for writing the submission.Source: Prompts · pp. 21–23
  • Table 1 prompt variants differed in budget wording: the 60-minute version added 45-minute and 50+-minute fallback guidance, while the no-budget version removed the time goal and management block.The remaining operating instructions were fixed across the four budgeted files and contained no time information.Source: Prompts · p. 23
Loading 2610.10833v1…