PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym

Jio Oh, Seunghyun Do, Young-Jun Lee, Steven Euijong Whang, Dongyeop Kang

arXiv:2609.37267v1cs.AI

TL;DR

Proactive agents could use idle compute to prepare assistance before users ask, but useful work can still be mistimed or misaligned. This paper proposes a 3T foundation, a design space, and PROACTIVITY-GYM; evaluations and a human study reveal coordination gaps and trust costs that task performance alone misses.

  • Problem

    Proactive work can create review costs and erode trust when it is irrelevant, poorly timed, or misaligned, so task success alone does not capture whether assistance is useful.

    Source: §1 · pp. 1–2

  • Method

    The paper connects task capability, temporal allocation, and trust to agent design and introduces PROACTIVITY-GYM for evaluating consequences across interactions.

    Source: §1 · pp. 2–3

  • Results

    Across 23 model–harness configurations, agents showed substantial limitations, while 30 participants showed sharp trust declines after misalignment and often preferred sleep-time help, even when imperfect.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support considering useful work, compute allocation, and evolving user trust together when designing and evaluating proactive agents.

    Source: §1 · p. 2

    PROACTIVITY-GYM is an initial dynamic testbed with simplified resource constraints and user models.

    Source: §6 · p. 10

Abstract

from arXiv · show

Proactive LLM agents can turn idle compute into useful support before users ask. Yet even correct work can misread user context, impose review costs, or undermine trust. This work proposes foundations for designing, realizing, and evaluating proactive LLM agents around three joint principles (3T): Task Capability, anticipating relevant needs and correctly performing useful work; Temporal Allocation, allocating compute according to resource availability and when results are needed; and Trust, sustaining users' confidence and appropriate reliance on the agent. We connect these objectives to a design space organized around five dimensions: task scope, anticipation horizon, activation trigger, processing timing, and intervention depth, and specify the situation and system modeling needed to support its choices, including user and environment representations, backbone LLMs, and agent harnesses. Lastly, we propose PROACTIVITY-GYM, a simulation-based evaluation testbed including multi-day scenarios, stateful environments, and persona-conditioned simulated users that can evaluate the consequences of proactive assistance across interactions. Evaluations across 23 model-harness configurations uncover substantial performance gaps across 3T and reveal that LLM judges often conflate task capability and trust. A human study with 30 participants demonstrates the importance of the joint 3T optimization: participants show sharp trust declines after intervention misalignment despite correct outcomes, and prefer sleep-time assistance, even when imperfect, to preserve ongoing focus. Together, these findings support designing and evaluating proactive agents through the joint consideration of useful work, compute allocation, and evolving user trust.

1 INTRODUCTION

The introduction argues that proactive agents should turn unused compute into useful assistance without imposing mistimed or misaligned work. It frames design around task capability, temporal allocation, and trust, and evaluates their consequences through PROACTIVITY-GYM and a human study.

1 INTRODUCTION

  • Proactive assistance can use otherwise idle compute to prepare support before users ask, but irrelevant, poorly timed, or low-quality work can impose review costs and erode trust.The introduction argues that additional compute can increase user utility only when proactivity is well designed.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The paper proposes three joint principles: task capability for anticipating and doing useful work, temporal allocation for scheduling compute, and trust for sustaining appropriate reliance.The principles distinguish useful work from when to perform it and how far to intervene.Source: §1 · p. 2
  • The foundation connects 3T to a five-dimension design space and the situation and system modeling needed to realize proactive agents.The specified components include user and environment representations, backbone LLMs, and agent harnesses.Source: §1 · pp. 2–3
  • Across 23 model–harness configurations, agents showed substantial limitations, particularly in deferring competing work; LLM-judge trust scores were relatively insensitive to intervention-depth misalignment.Task-capability scores correlated weakly with temporal-allocation and intervention-depth alignment (r = 0.31 and 0.34).Source: §1 · p. 2
  • Among 30 participants, a single intervention-depth misalignment sharply reduced trust despite correct outcomes, while participants often accepted sleep-time assistance even when it required correction.Participants also favored deferral when immediate help posed no resource or deadline conflict, preserving ongoing focus.Source: §1 · p. 2

2 RELATED WORK

The related work links proactive-agent design to complementary research on initiative, compute scheduling, and trust. It also identifies a gap in evaluating these objectives together across evolving interactions.

2 RELATED WORK

  • Prior work on mixed-initiative assistance, trust-aware dialogue, and idle- or sleep-time compute provides complementary foundations that this paper unifies through 3T.The paper connects those perspectives to design choices and system requirements.Source: §2 · p. 3
  • The paper translates this combined perspective into agent design choices and system requirements rather than treating the prior research strands as isolated concerns.The cited strands include mixed-initiative research, trust-aware dialogue, and anticipated-need computation.Source: §2 · p. 3
  • Existing benchmarks separately test need prediction, intervention timing, corrective actions, or personalized assistance, while PROACTIVITY-GYM evaluates TC, TA, and TR jointly across multi-day interactions.The related work also notes interactive benchmarks for personalized assistance and evolving tasks.Source: §2 · p. 3

3 PRINCIPLES OF PROACTIVITY IN LLM AGENTS

The paper defines proactivity as anticipating user needs without an explicit request, then separates its design into task capability, temporal allocation, and trust. It connects these objectives to joint design choices and a weighted formulation.

3 PRINCIPLES OF PROACTIVITY IN LLM AGENTS

  • Proactivity is anticipatory behavior intended to address user needs, opportunities, or problems without an explicit request.The paper distinguishes initiative from useful assistance by assessing benefits, costs, and uncertainties through the 3T objectives.Source: §3 · p. 3

3.1 THREE DESIGN OBJECTIVES (3T)

  • Task capability combines identifying relevant user needs with correctly performing useful work to address them.Failure can arise either from pursuing work the user does not value or from producing an inadequate result for a relevant need.Source: §3.1 · p. 3
  • Temporal allocation schedules compute according to resource availability and when results are needed, including both when work is performed and when it is presented.Work that is not worth an immediate interruption may still be useful if prepared with spare compute and delivered later.Source: §3.1 · p. 3
  • Interaction time is active workflow time when proactive work may compete for resources; sleep time is inactive time when background work avoids that competition.Brief idle intervals within an active phase still count as interaction time; literal sleep is an intuitive example, not a requirement.Source: §3.1 · p. 4
  • Trust is users’ confidence in and willingness to rely on an agent, while intervention depth can serve as a context-dependent behavioral proxy for trust.Delegation should be interpreted against the user’s preferences, task, and interaction history; changing from autonomous edits to review may signal reduced trust.Source: §3.1 · p. 4

3.2 JOINT DESIGN OBJECTIVE

  • Prior work often evaluates task or intervention success without separately measuring trust, and timing evaluations may omit compute allocation.The paper frames the joint formulation as limited for LLM agents, despite related work addressing complementary aspects.Source: §3.2 · pp. 4–5
  • The design space organizes proactive assistance around five dimensions that inform what work to pursue, when to initiate and process it, and how autonomously to proceed.The 3T labels indicate which objectives inform the design choices.Source: Figure 3 · p. 5
  • A weighted formulation scores agent designs across task capability, temporal allocation, and trust, with weights representing each objective’s relative importance.Joint consideration matters because useful work may offer little benefit if it competes with ongoing tasks or undermines trust.Source: §3.2 · p. 5

4 TECHNICAL LAYERS

The technical layers translate 3T into five proactive-action choices and specify situation and system modeling to support context-sensitive decisions.

4.1 PROACTIVITY DESIGN SPACE

  • The five dimensions specify what work to pursue, when to initiate and process it, and how far the agent should proceed.They cover task scope, anticipation horizon, activation trigger, processing timing, and intervention depth.Source: §4.1 · p. 5

4.2 SYSTEM REALIZATION

  • Situation modeling combines interaction history, memory, and observations to track user goals, attention, preferences, trust, task progress, deadlines, and resources.It distinguishes observed facts from implicit states inferred from behavior, updating both as new evidence arrives.Source: §4.2 · p. 6
  • The backbone LLM anticipates and performs useful work, while the harness coordinates tools, memory, permissions, pending tasks, compute, and intervention depth.The harness tracks work across interaction and sleep periods and allocates it around workloads and deadlines.Source: §4.2 · p. 6
  • The decision loop builds a situation representation, selects an action from that representation and context, then updates context with action traces and new observations.Actions can start, continue, defer, or present work; allocate compute; apply results; or do nothing.Source: §4.2 · p. 6, Figure 4 · p. 6

5 PROACTIVITY-GYM

PROACTIVITY-GYM evaluates agents in dynamic, multi-day scenarios, while a human study examines how task performance, timing, and intervention alignment shape user judgments. Results reveal gaps across objectives and show that correct work alone does not ensure acceptance or trust.

5 PROACTIVITY-GYM

  • PROACTIVITY-GYM combines a simulated clock, stateful tool environments, and persona-conditioned users whose feedback and trust state respond to interactions.Its 10 scenarios span 7–10 simulated days and include latent needs, constraints, action-dependent events, and cases where no intervention is needed.Source: §5 · pp. 6–7, Figure 5 · p. 7

5.1 EXPERIMENT SETUP & EVALUATION

  • The evaluation tested nine models across three harnesses, using three runs per scenario-persona pair and measures of task capability, temporal allocation, and trust.Metrics include TC, TA, intervention-depth agreement (TR-D), and history-based trust ratings (TR-J).Source: §5.1 · p. 7

5.2 RESULTS

  • 65.1 average TC was Claude Opus 5’s highest score, alongside 51.7% TA and 52.4% TR-D; other models scored below 20% TA, with TR-D from 39% to 51%.The evaluation covered 23 model–harness configurations.Source: §5.2 · pp. 7–8, Figure 6 · p. 8
  • Across five open models, switching from Claude Code to OpenClaw lowered TC by 3.6 points on average, while TC correlated more strongly with TR-J than with TA or TR-D.The reported correlations were r = 0.70 for TC with TR-J, r = 0.31 with TA, and r = 0.34 with TR-D.Source: Figure 6 · p. 8, §5.2 · p. 8
  • LLM judges gave some agents high trust ratings despite low intervention-depth agreement, suggesting they can overlook misalignment that human participants notice.Claude Opus 5 scored 4.72 and 4.60 on two trust dimensions while scoring 52.4% for TR-D; participants’ trust dropped sharply after a misaligned intervention despite correct outcomes.Source: §5.2 · p. 8, §5.3 · p. 9

5.3 HUMAN STUDY

  • Acceptance rose from 26.7% for correct assistance competing with ongoing tasks to 97.8% for sleep-time assistance requiring revision; trust losses after misalignment exceeded gains after realignment.The reported trust change was −1.86 points for A→M versus +1.27 for M→A, and an A-M-A sequence did not restore initial trust.Source: Figure 7 · p. 9, §5.3 · p. 9

6 CONCLUSION

The conclusion argues that proactive agents need joint consideration of task capability, temporal allocation, and trust, and that the proposed gym remains an initial testbed with simplified constraints and user models.

6 CONCLUSION

  • Task performance alone is insufficient: useful proactive assistance must account for when work happens and how far the agent intervenes.The authors conclude that current agents struggle to coordinate all three objectives and call for further study of joint 3T optimization.Source: §6 · p. 10
  • PROACTIVITY-GYM currently uses simplified resource constraints and user models; longer-term evaluation in real-world settings remains future work.Source: §6 · p. 10

Compute and trust foundations

A EXTENDED RELATED WORK

  • The paper connects compute allocation, preference-sensitive intervention, and cognitive trust constructs to behavioral design choices and system requirements.Source: A EXTENDED RELATED WORK · p. 17

A.1 RESEARCH LANDSCAPE ACROSS THE 3T OBJECTIVES

  • The comparison treats benchmark coverage as scope rather than performance and gives partial credit for request prediction, fixed pre-query computation, or simple intervention preferences.Its stated criteria distinguish explicit, partial, and absent coverage across the 3T objectives.Source: §A.1 · p. 17, Table 2 · p. 18

B FIVE CONSTRUCTS OF TRUST

  • The adopted trust model separates cognition-based trust—understandability, competence, and reliability—from affect-based trust—personal attachment and faith.Source: Compute and trust foundations · p. 19

C ADDITIONAL MOTIVATIONAL SCENARIOS

  • The scenarios illustrate how task scope, trigger, timing, and intervention depth can vary: meeting preparation, an additional code repair, and overnight paper preparation.The overnight example separates processing timing from when the anticipated result is needed.Source: C ADDITIONAL MOTIVATIONAL SCENARIOS · p. 19

D.1 SCENARIOS AND TASKS

PROACTIVITY-GYM uses multi-day scenarios, persona-specific policies, and changing task conditions to evaluate task capability, temporal allocation, and trust. Its evaluation scores complete runs while separately measuring task work, prioritization, and intervention-depth alignment.

D.1 SCENARIOS AND TASKS

  • Ten scenarios span research, travel, living, food, career, business, education, finance, shopping, and health, with goals, timelines, records, tools, availability, and three delegation-style personas.Reviewer requires explicit approval before changes, Planner permits preparation within the stated task, and Operator delegates specified actions.Source: §D.1 · p. 19, Figure 8 · p. 20
  • Each scenario has 10–19 tasks with triggers, timing, evidence, outputs, dependencies, latent needs, resource conflicts, and persona-specific approval requirements.Task specifications define the criteria used to assess TC, TA, and TR; the scenarios and rubrics were manually authored and then refined with LLMs.Source: §D.1 · pp. 19–20

D.2 USER SIMULATOR

  • A deterministic state machine approves, declines, or defers proposed actions using requested actions, time, persona, prior decisions, and trust state; the simulator conveys that decision without granting extra permission.Messages are not delivered and approval requests receive no decision during scheduled sleep or other unavailable intervals.Source: §D.2 · pp. 20–21

D.3 EXAMPLE SCENARIO

  • The Monitor Purchase example links tasks across a ten-day shopping scenario: agent-scheduled stock checks and later delivery or price tasks depend on earlier actions and user policy.The personas differ in whether they permit saving a plan or placing a £160 order without approval; replacement and price-adjustment claims require approval for all.Source: §D.3 · p. 21, Table 3 · p. 22

E.1 EVALUATION UNIT

  • Each configuration is evaluated on 90 complete multi-day runs: 10 scenarios, three personas, and three repetitions per scenario-persona pair; only tasks whose activation conditions occur count.The ten scenarios contain 124 tasks, but conditional branches can leave some tasks inactive in a run.Source: §E.1 · p. 21

E.2 TASK CAPABILITY ( TC )

  • TC scores task requirements at the agent’s chosen intervention depth by averaging applicable components, with rule-based checks and LLM judgments assessing different aspects of the work.Components include evidence, outputs, constraints, semantics, and completion; permission, timing, and intervention-depth appropriateness are evaluated separately.Source: §E.2 · pp. 21, 23

E.3 TEMPORAL ALLOCATION ( TA )

  • TA tests whether agents prioritize the urgent task and explicitly defer its competitor; a run earns credit only when both decisions are correct.Scenarios specify competing tasks, deadlines, shared resources, and later opportunities; continuation-plan quality is reported separately and does not affect the main score.Source: §E.3 · pp. 23–24

E.4 TRUST ( TR )

  • Trust is measured separately as task-level depth agreement (TR-D) and judges’ 1–5 ratings of five trust constructs (TR-J); those ratings estimate perceived interaction quality, not psychological trust directly.Expected intervention depth depends on persona, task, and applicable approvals or denials.Source: §E.4 · p. 24

E.5 LLM-AS-A-JUDGE

Judge scores track configuration rankings closely but differ in calibration and some temporal-allocation decisions. More reasoning improves task capability and temporal allocation in the tested GPT 5.6 Sol setup, while trust metrics change little.

E.5 LLM-AS-A-JUDGE

  • The judges score frozen evaluation inputs with model and harness identities withheld; TC and TA calls or quotes are checked against trajectories, and TR-J ratings require log evidence.Scores average two independent judge models, Qwen 3.8-27B and Gemini 3.8-Flash.Source: §E.5 · p. 24
  • On TA, judges agree on 89.7% of exact NOW/LATER decisions, but Gemini marks NOW correct more often and gives fewer fully correct NOW/LATER decisions.Gemini’s mean TR-J score is 0.134 points lower than Qwen’s, although the judges are strongly correlated on TC and TR-J.Source: §E.5 · p. 25, Table 5 · p. 25
  • After aggregation by configuration, judge correlations are .993 for TC, .950 for TA, and .980 for TR-J; TC rankings correlate at ρ = .999.Judge choice has a larger effect on absolute scores than on relative configuration rankings.Source: §E.5 · p. 25, Table 6 · p. 25

F.1 COMPLETE RESULTS

  • Each configuration specifies a harness, backbone model, and reasoning effort; the evaluation covers 23 configurations, with 10 scenarios, 3 personas, and 3 repetitions per scenario-persona pair.Open-weight models use Codex, Claude Code, and OpenClaw; GPT-family models use Codex and OpenClaw, and Claude-family models use Claude Code and OpenClaw.Source: §F.1 · p. 25, Table 7 · p. 26

F.2 DOES MORE REASONING IMPROVE PROACTIVITY?

  • TC rises from 49.96 (no reasoning) to 58.93 (extra-high), and TA from 10% to 17.2%, while TR-D shifts from 51.2% to 53.9% and TR-J from 3.99 to 4.06.The experiment varies reasoning effort for GPT 5.6 Sol with Codex.Source: §F.2 · p. 25

F.3 DETAILED RESULTS ON TASK CAPABILITY

Task-capability results distinguish producing semantically sound content from completing all task requirements, with additional weaknesses in evidence acquisition and content constraints.

F.3 DETAILED RESULTS ON TASK CAPABILITY

  • Claude Opus 5 scores 58.13 on Completion versus 74.67 on Semantics, while GPT 5.6 Sol scores 34.93 versus 67.04, respectively.The results suggest correct, grounded content is easier to produce than content fulfilling the full task requirements.Source: §F.3 · p. 27
  • Evidence acquisition scores are only 61.7–62.7 for GPT 5.6 Sol, Claude Sonnet 5, and Claude Opus 5; content-constraint satisfaction ranges from 48.8 for Sol to 76.7 for Opus.The rule-based components expose limitations in satisfying task requirements.Source: §F.3 · p. 27
  • Across the reported criteria, scores consistently increase with model size within each model family.Source: §F.3 · p. 27

F.4 DETAILED RESULTS ON TEMPORAL ALLOCATION

Temporal-allocation results show a gap between selecting urgent work and explicitly deferring competing work, with continuation quality also varying sharply by model.

F.4 DETAILED RESULTS ON TEMPORAL ALLOCATION

  • 44.32% of cases select the correct immediate task, compared with 10.8% that explicitly defer competing work, a gap persisting across configurations.Higher later scores generally coincide with better continuation quality.Source: §F.4 · p. 27
  • Claude Opus 5 scores 50.69 on continuation quality, compared with 0.28 to 15.69 for other models.Continuation quality evaluates the concreteness and feasibility of a plan for deferred work.Source: §F.4 · p. 27

F.5 DETAILED RESULTS ON TRUST ( TR-J )

Trust scores can mask mismatches between an agent’s intervention depth and user preference, while scenario results show variation in trust alignment across scenarios and personas. The human-study scenarios examine trust over repeated interactions and compare trade-offs involving intervention depth, timing, and task capability.

F.5 DETAILED RESULTS ON TRUST ( TR-J )

  • LLM judges rate understandability and technical competence highly but are relatively tolerant of intervention-depth misalignment; humans penalize such misalignment more strongly.The results also show relatively lower ratings for reliability or personal attachment.Source: §F.5 · p. 27
  • When users prefer execution, models end without a valid execution attempt in 77.21% of cases; this occurs in 45.96% for Claude Opus 5 despite its TR-J score of 4.25.This indicates frequent under-intervention relative to delegated execution preferences.Source: §F.5 · p. 27

F.6 SCENARIO AND PERSONA VARIATION

  • Across scenarios, TC ranges from 32.45 to 58.31 and TR-D from 28.10% to 62.82%, while TA reaches at most 20.05%.TR-D varies more across personas; agents align better with review-preferring personas than with those preferring delegated execution.Source: §F.6 · p. 31, Figure 11 · p. 31

G DETAILS ON HUMAN STUDY

  • The human study recruits 30 LLM-familiar participants and uses 14 scenarios, including week-long trust scenarios and pairwise comparisons involving TR, TA, and TC.Participants assess agent behavior after reading interaction logs from a user perspective under resource constraints.Source: G DETAILS ON HUMAN STUDY · p. 31

G.1 QUESTION TYPE 1: VARYING TRUST ACROSS INTERACTIONS

  • Four simulated week-long logs vary intervention alignment while keeping task content and final outcomes correct; participants rate trust and intervention appropriateness on Days 2, 4, and 7.The scenarios cover study scheduling, grocery shopping, delivery scheduling, and workshop preparation.Source: §G.1 · p. 32, Figure 12 · p. 33

G.2 QUESTION TYPE 2: TRUST PREFERENCE

  • Four scenarios compare aligned versus misaligned intervention depth at equal content quality and perfect content with misaligned depth versus imperfect content with aligned depth.Participants choose between agents and rate intervention level and content appropriateness.Source: §G.2 · p. 32, Figure 13 · p. 34

G.3 QUESTION TYPE 3: TEMPORAL ALLOCATION PREFERENCE

  • Three resource-sharing scenarios compare immediate assistance that delays work or consumes needed resources with deferred assistance scheduled around sleep time and availability.Further comparisons vary review or adjustment timing and contrast well-allocated imperfect work with high-quality work that interrupts focus.Source: §G.3 · p. 32, Figure 14 · p. 34

G.4 QUESTION TYPE 4: TASK CAPABILITY PREFERENCE

  • Three scenarios ask participants to compare agent responses with and without proper task capability, rating which better fits the stated situation.The scenarios cover shopping recommendations, production and shipping scheduling, and recipe-based meal planning.Source: §G.4 · p. 32, Figure 15 · p. 35

G.5 DETAILED SURVEY RESULTS

Survey results show that people value accurate content and intervention-depth alignment, while also accepting imperfect work when it is deferred to preserve focus. Qualitative feedback reinforces the need to jointly consider task capability, temporal allocation, and trust.

G.5 DETAILED SURVEY RESULTS

  • Survey error bars use 95% confidence intervals estimated from 20,000 bootstrap resamples.Source: §G.5 · p. 36
  • Accurate actions or content earned 2.44 points more (CI: [2.04, 2.82]); aligned intervention depth earned 2.16 points more in appropriateness (CI: [1.63, 2.62]).When quality and alignment conflicted, participants chose incomplete but aligned assistance in 68.3% of comparisons when approval was required, versus 15.0% when execution was delegated.Source: §G.5 · p. 36, Figure 16 · p. 36
  • Sleep-time allocation increased acceptance by 67.7 percentage points (95% CI: [52.2, 82.2]) when immediate help competed with users’ compute.63 of 90 responses rejected immediate help but accepted sleep-time help; 97.8% accepted imperfect sleep-time work requiring an average 20 minutes of correction.Source: §G.5 · p. 36
  • Even without resource or deadline conflicts, participants preferred sleep-time assistance in 64.4% of cases when review took 10 minutes.Participants’ comments also identify preserving current focus as a reason for sleep-time allocation.Source: §G.5 · p. 37, §G.6 · p. 37
  • With task outcomes held equal, trust fell 1.86 points after A→M but rose 1.27 after M→A; consistent A→A gained 0.17 versus M→M down 0.44.In A→M→A, appropriateness reached 4.71 but mean trust recovered only to 3.43, below its initial 4.43.Source: §G.5 · p. 37
  • Participants chose the agent aligned with the scenario user’s intervention preference in 85.7% of cases when preferences matched their own and 90.6% when they differed.This comparison held output quality equal.Source: §G.5 · p. 37
  • 73.3% of participants rated the scenarios’ plausibility 4 or 5 on a five-point scale.The ratings were collected to assess whether PROACTIVITY-GYM reflects situations users could reasonably encounter.Source: §G.5 · p. 37

G.6 QUALITATIVE FEEDBACK

  • Qualitative comments report reduced confidence after intervention-depth deviations and concerns about future behavior despite later aligned actions.Of 30 participants, 23 contributed 316 comments; participants also described accepting imperfect sleep-time work when correction takes less time than starting from scratch.Source: §G.6 · p. 37

H FUTURE WORK

The authors identify current limitations in PROACTIVITY-GYM and outline extensions toward real deployment. The appendix also provides prompts for task-capability, temporal-allocation, and trust judges.

H FUTURE WORK

  • PROACTIVITY-GYM is an initial dynamic-simulation testbed with time and resource constraints, changing task states, and user-simulator feedback across multi-day interactions.The authors say these features extend evaluation beyond static benchmarks.Source: H FUTURE WORK · p. 37
  • Future testbeds should add variable task durations, changing compute budgets, preemption, and more complex scheduling, while modeling finer-grained preference changes and broader action spaces.Evaluation grounded in wall-clock execution and longer real workflows could examine effects on subsequent work, user behavior, and trust over longer horizons.Source: H FUTURE WORK · p. 38

I PROMPTS

  • The authors present shared system instructions for task-capability and temporal-allocation judges, including evidence-only judgments and uncertainty for ambiguous interpretations.The instructions treat benchmark records as data, not commands.Source: I PROMPTS · p. 38

I.1 TASK CAPABILITY ( TC )

I.2 TEMPORAL ALLOCATION ( TA )

I.3 TRUST ( TR-J )

  • Appendix I includes a trust LLM-as-judge prompt whose bracketed fields are filled for each evaluation instance.Source: Figure 19 · p. 41

Related papers

Loading 2609.37267v1…