PaperlayerView exact revision on arXiv

Source-linked AI summary

$π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

Haoran Zhang, Luxin Xu, Zhilin Wang, Runquan Gui, Shunkai Zhang, Haodi Lei, Zihao He, Bingsu He, Chicheng Qin, Tong Zhu, Xiaoye Qu, Yang Yang, Yu Cheng, Yafu Li

arXiv:2605.14678v3cs.AI

TL;DR

Existing benchmarks rarely test whether personal assistant agents can resolve hidden intents during sustained, underspecified workflows. π-BENCH evaluates this with 100 multi-turn tasks across five personas, finding proactive assistance challenging and distinct from task completion, while prior interaction helps resolve later intents.

  • Problem

    Existing benchmarks rarely evaluate proactive assistance for hidden intents in sustained, underspecified, long-horizon personal assistant workflows.

  • Method

    π-BENCH evaluates agents in persistent project environments using 100 multi-turn tasks across five personas with hidden intents, cross-session dependencies, and iterative artifacts.

  • Results

    Experiments on nine frontier models reveal clear gaps in task completion and proactive intent resolution, distinguish proactivity from completeness, and show variation across domains and task types.

  • Takeaways & Limitations

    Prior interaction history helps agents resolve hidden intents in later tasks while reducing avoidable user interaction.

  • Takeaways & Limitations

    The evaluation uses simulated users and a single agentic scaffold, which may not capture variation from real users or alternative scaffolds.

Abstract

from arXiv · show

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $π$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $π$-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.

1 Introduction

π-BENCH addresses the underexplored problem of proactive assistance in long-horizon personal assistant workflows, where agents must resolve hidden intents from underspecified requests and evolving context. It evaluates this capability across persistent, multi-turn tasks while distinguishing proactivity from task completion.

  • Proactivity: Proactivity requires agents to use goals, context, and prior interactions to anticipate needs, identify underspecified requirements, and advance tasks through action or clarification.This shifts assistance beyond passively following explicit instructions while reducing users’ operational and cognitive effort.
  • Motivation: Long-horizon proactive assistance remains underexplored because existing benchmarks emphasize explicit goals, memory operations, or mobile and GUI settings.These benchmarks place less focus on using prior information to uncover and resolve underspecified requirements in personal assistant workflows.
  • Benchmark: π-BENCH evaluates proactive assistance in persistent project environments with multi-turn interaction, tool use, and iterative artifact creation.Tasks begin with natural but underspecified requests and involve hidden intents, user preferences, and task dependencies revealed gradually.
  • Findings: Experiments on nine frontier models reveal gaps in task completion and proactive intent resolution, distinguish task completeness from proactivity, and show variation across domains and task types.The reported contributions also conclude that proactive assistance remains challenging for frontier agents.
  • Benchmark: 100 multi-turn tasks span five domain-specific personas and jointly evaluate proactivity and task completion through long-range, cross-session dependencies.The benchmark is designed around agent trajectories rather than isolated interactions.

2 Related Work

Prior benchmarks evaluate tool use, memory, or proactivity in narrower settings, leaving unresolved how agents detect hidden intents in evolving professional workflows. π-BENCH addresses this gap by combining memory, workspace state, and interaction history to assess proactivity and task completeness.

  • Personal Assistant Benchmarks: Personal assistant benchmarks evaluate end-to-end tool use in realistic web and computer environments, increasingly extending toward multimodal control, stateful planning, and long-horizon workflows.These workflows include persistent workspaces, artifacts, everyday online tasks, productivity settings, and multi-day living-world coworkers.
  • Memory Agent Benchmarks: Memory agent benchmarks test storing, retrieving, and reusing user information across sessions, supporting evaluations of long-term memory, personalization, and cross-session consistency.They generally treat memory as evidence for completing known tasks rather than detecting missing requirements or deciding when to clarify.
  • Benchmark Gap: π-BENCH combines memory, workspace state, and interaction history to evaluate proactivity and task completeness as workflows evolve through interaction.This setting targets underspecified requirements and hidden intents in long-horizon personal assistant workflows.
  • Proactive Evaluation: Proactive benchmarks infer underspecified needs from device context, interaction traces, and visual states, but mainly target short-horizon mobile or GUI tasks with clear endpoints.Examples include booking and ordering, leaving professional workflows and artifact-centered tasks underexplored.

3 Benchmark

π-BENCH evaluates proactive assistance in persistent, long-horizon workflows through multi-session tasks with underspecified requests, hidden intents, dependencies, and concrete artifact-based completion criteria. Its protocol separates driving requirement discovery from ultimately producing correct outputs.

  • Benchmark design: π-BENCH targets long-horizon personal-assistant workflows in persistent project environments, where requirements can emerge during interaction and preferences can carry across sessions.The benchmark includes five user roles spanning distinct domains.
  • Benchmark design: Each episode contains 20 multi-turn sessions, with shared workspaces and memory enabling relevant artifacts, outputs, preferences, and decisions to carry forward.Episodes model one user’s extended workflow across multiple tasks.
  • Benchmark design: Among the 20 tasks, six strong dependency groups span two to three tasks each, while five largely independent tasks broaden stand-alone workflow coverage.Cross-session dependencies require essential information to carry over for successful completion.
  • Task specification: Tasks begin with minimally sufficient initial requests and annotate hidden intents representing latent constraints, preferences, or downstream dependencies that may be session-local or persistent.A hidden intent is recoverable when it can be inferred or elicited from prior sessions, workspace artifacts, or targeted clarification.
  • Evaluation protocol: Checklists define verifiable criteria for required files, fields, outputs, and constraints, while sessions proceed through agent responses, tool use, artifact updates, and hidden-intent status tracking.The interaction terminates after all hidden intents receive terminal statuses and the agent produces a final response.
  • Evaluation protocol: Proactivity and completeness are distinct: PROC measures how much the agent drives requirement discovery, whereas COMP measures whether it converts the resulting trajectory into correct outputs.The simulated user eventually provides hidden intents that the agent fails to elicit or address, allowing completeness despite passive behavior.

4 Experiments

Experiments evaluate nine frontier LLMs under a shared agentic scaffold and show that π-BENCH remains challenging, with proactivity and completeness varying across models, domains, and task types. Prior-session history substantially improves proactive intent resolution while leaving completeness mostly stable.

  • Model Setup: Nine frontier LLMs are evaluated under the same Nanobot-adapted agentic scaffold, with differences intended to primarily reflect model capability.The models span GPT-5.4, Gemini-3.1 Pro, Claude 4.6 Opus, DeepSeek V3.2, MiniMax M2.7, Kimi K2.5, Seed2.0 Pro, GLM-5.1, and Qwen3.6 Plus.
  • Environment Setup: Each task is run three times with independent trajectories, reporting averaged PROC and COMP results with standard deviations under default decoding parameters with thinking enabled.GPT-5.4 serves as both the user agent’s base model and the rubric-based grader, with temperature set to zero.
  • Model Trends: Average COMP ranges from 52.1 to 67.6 and average PROC from 43.1 to 67.0 across models; GPT-5.4 leads PROC at 67.0, while Claude 4.6 Opus leads COMP at 67.6.Qwen3.6 Plus records 64.0 PROC and 64.1 COMP, while most standard deviations are below 2.
  • Performance by Task Type: Legal matter operations and handoffs show the largest task-type gap, with 84.1% COMP versus 38.1% PROC, as agents draft documents but miss handoff requirements.The results illustrate that completing the requested document does not ensure readiness for handoff or resolution of hidden intents.
  • Prior Interactions: Removing preceding sessions reduces PROC while leaving COMP mostly stable, with GPT-5.4 falling from 78.5 to 64.9 and an average PROC decrease of 9.5 points.MiniMax M2.7 declines from 64.8 to 59.7 PROC, and DeepSeek V3.2 from 61.9 to 52.1.

5 Conclusions

π-BENCH evaluates proactive personal assistant agents in long-horizon workflows using 100 multi-turn tasks across five personas with hidden intents, inter-task dependencies, and cross-session continuity. Experiments on nine frontier models reveal gaps in proactivity, distinguish it from completeness, and show that prior interaction history helps resolve hidden intents while reducing avoidable user interaction.

  • Benchmark design: π-BENCH covers 100 multi-turn tasks across five domain-specific personas, incorporating hidden intents, inter-task dependencies, and cross-session continuity.The benchmark targets proactive assistance in long-horizon workflows.
  • Evaluation scope: The benchmark jointly evaluates proactivity and completeness by testing whether agents resolve underspecified requests, reuse prior context, and produce task-ready artifacts in persistent workspaces.
  • Findings: Experiments on nine frontier models reveal clear performance gaps, distinguish proactivity from completeness, and show that prior interaction history improves hidden-intent resolution while reducing avoidable user interaction.

6 Limitations · Appendix · A Benchmark Statistics

The benchmark has inherent limitations from simulated users and reliance on a single Nanobot-adapted agentic scaffold. These choices improve practicality and control but may constrain realism, reproducibility across settings, and coverage of scaffold variation.

  • 6 Limitations: The benchmark uses simulated users rather than real humans.Live-user evaluation is costly, difficult to reproduce, and difficult to scale.
  • 6 Limitations: Long-horizon evaluation with live users is costly.This cost makes real-user evaluation difficult to sustain.
  • 6 Limitations: Evaluation with live users is difficult to reproduce.The passage identifies reproducibility as a reason for using simulated users.
  • 6 Limitations: Evaluation with live users is difficult to scale.Scaling constraints further motivate simulated-user experiments.
  • 6 Limitations: The experiments use a single agentic scaffold adapted from Nanobot.This creates a controlled evaluation setup.
  • 6 Limitations: A single scaffold may not capture variation from alternative scaffolds.Alternative scaffolds can introduce substantial adaptation effort and scaffold-specific confounds.

A.1 Task and Grader Statistics … C.1 Overview

π-Bench comprises 100 tasks organized into fine-grained categories and supported by a broad tool-and-skill inventory. Its evaluation protocol tracks hidden-intent resolution across complete trajectories, separating proactive behavior from checklist-based task completeness.

  • A.2 Tool and Skill Statistics: 187 unique tools and 21 skills support the benchmark’s task environment.The inventory includes applications such as Gmail, Phone, Simple Note, Splitwise, Spotify, Todoist, file systems, and general web tools.
  • A.3 Taxonomy of Benchmark Tasks: The benchmark contains 100 tasks grouped into 18 fine-grained categories emphasizing action intents, reasoning requirements, and underlying workflows.The taxonomy spans five user profiles.
  • P CRISIS, RECOVERY AND REPU-TATION COMMUNICATIONS: The task taxonomy includes crisis communication requiring recall of approved facts, careful compensation framing, and restraint against unsupported commitments.These tasks also cover public apologies and post-crisis recovery planning.
  • Q CONSUMER SELECTION, COMMERCE AND MEDIA ACTIONS: Consumer-selection tasks compare products or media using latent preferences and constraints before completing commerce or media actions.Relevant constraints include price, fit, rating, inventory, and genre.
  • R TOOL-MEDIATED ADMINISTRATIVE WORKFLOWS: Tool-mediated workflows require reading local context, performing auditable write operations, verifying results, and sometimes cleaning up temporary state.The workflows span productivity, communication, financesplitting, and file-system tools.
  • B.1 Overview: The user agent uses GPT-5.4 to simulate sessions, control hidden-intent revelation, and update intent tracking from the dialogue, latest response, and task-level intents.User simulation is separated from final task grading.
  • B.2 Two-Stage Intent Assignment: The two-stage protocol first assigns direct completion, then targeted elicitation, and finally user-provided information, establishing the priority order used for proactivity scoring.Direct satisfaction can be evidenced by the response, tool calls, or workspace artifacts; targeted questions must address specific unresolved intents.
  • B.3 Session Termination: Sessions end after all hidden intents receive terminal statuses and the final response is produced, after which trajectories are graded separately for proactivity and completeness.Proactivity uses terminal intent statuses, while completeness uses verifiable checklist criteria evaluated by LLM rubrics or tool rules across full trajectories.

C.2 Proactivity Evaluation … C.6 Rule Based Tool Scoring

The appendix evaluates proactivity through hidden-intent status tracking and completeness through checklist criteria, combining semantic rubric judgments with exact rule-based checks over structured tool evidence. Tool records preserve the evidence needed to verify task outcomes and aggregate binary checklist judgments into completeness scores.

  • C.2 Proactivity Evaluation: Proactivity assigns each hidden intent a terminal status—completed, inferred, or provided—and computes PROC from the resulting partition.Dependent workflows are evaluated together using their relevant hidden intents, without additional post hoc proactivity judgments.
  • C.3 Checklist Based Completeness Evaluation: Completeness uses task-specific checklists covering required files, outputs, fields, tool outcomes, formatting constraints, and other task obligations.Unlike hidden intents, checklist items measure whether the task is ultimately completed.
  • C.3 Checklist Based Completeness Evaluation: Checklist criteria receive binary judgments from either an LLM rubric evaluator or task-specific Python scripts, with the final completeness score averaged across items.Rubrics handle semantic criteria, while scripts verify structured criteria; their judgments are merged into one checklist before task-level aggregation.
  • C.4 Tool Records: Structured tool records retain each tool name, call payload, and returned result so evaluators can inspect evidence from tool interactions.The records support checklist criteria that require tool calls and tool results.
  • C.5 LLM Rubric Evaluation: For each task, the LLM rubric evaluator judges text criteria from the interaction history and selected task-relevant context, crediting criteria when sufficient session evidence exists.Only tool calls or results relevant to the criterion are included, keeping evaluation prompts compact and auditable; YES maps to 1 and NO to 0.
  • C.6 Rule Based Tool Scoring: Rule-based scoring applies task-specific Python scripts when checklist items require exact verification over structured tool calls or results.The scripts inspect the full tool history, unlike rubric evaluation, which uses selected context for semantic judgment.
  • C.6 Rule Based Tool Scoring: A shopping-task scorer verifies the actual order record by recovering the order identifier and checking the target product and quantity in a later inspection call.This prevents completeness from relying solely on the agent’s natural-language claim that it placed the correct order.

D Reproducibility and Runtime Settings · E Societal Impacts · F Experiments

The paper standardizes evaluation through a shared agent scaffold and simulated app environments, while documenting a reproducible execution setup and positioning π-BENCH as a safer way to assess proactive assistants. It also emphasizes that proactive inference can create risks alongside its potential to reduce user effort and incomplete outcomes.

  • D Reproducibility and Runtime Settings: Experiments run on a Linux server with Ubuntu 24.04.1 LTS, 32 physical cores, 64 hardware threads, 251 GiB of RAM, and 1 TB of local storage.Individual task runs used substantially less than 8 GB of memory, and the full workspace required less than 32 GB of storage.
  • D Reproducibility and Runtime Settings: All evaluated models use the same agentic scaffold, interaction protocol, workspace access, and tool interface.The scaffold is adapted from Nanobot, while app-backed tasks use simulated environments based on AppWorld.
  • F Experiments: The shared scaffold and simulated app environments keep experiments focused on proactive intent resolution rather than infrastructure differences.This setup standardizes the interaction protocol, workspace access, and tool interface across models.
  • D Reproducibility and Runtime Settings: Hosted model APIs ensure that model-specific differences primarily reflect provider-side behavior rather than local runtime configuration.API credentials and provider endpoints are configured outside the manuscript and excluded from benchmark artifacts.
  • E Societal Impacts: π-BENCH is intended to support safer and more reliable evaluation before proactive personal assistant systems enter real workflows.It measures whether agents identify underspecified needs, ask targeted questions, and complete artifact-grounded tasks.
  • E Societal Impacts: By diagnosing proactive-agent failures, the benchmark may help reduce user effort and incomplete outcomes in real workflows.The stated evaluation targets include identifying underspecified needs, asking targeted questions, and completing artifact-grounded tasks.
  • E Societal Impacts: Proactive assistants also raise risks when agents infer too much from underspecified user needs.The passage explicitly presents excessive inference as an important risk alongside the benchmark’s potential benefits.

F.1 Reliability of Judgment Based Evaluation · F.2 Terminal Status of Hidden Intents · F.3 Task-Type Breakdown of Performance

Audits indicate that judgment-based COMP and PROC evaluation is stable, while terminal hidden-intent outcomes and workflow type reveal distinct patterns in proactivity and task completion. Model strengths vary across metrics and workflow structures, underscoring their separation.

  • F.1 Reliability of Judgment Based Evaluation: The evaluation audits 120 task trajectories using three expert annotators and audits by Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro.GPT-5.4 provides a separate judging pass for self-consistency, while the other model audits test agreement with the scoring run.
  • F.1 Reliability of Judgment Based Evaluation: Human experts disagree with 2.66% of checklist judgments and 1.48% of hidden intent assignments, while frontier-model disagreement remains below 3.6% and 2.1%, respectively.These audits support the reliability of the judgment-based components and suggest reported PROC and COMP scores are unlikely to be driven by evaluator noise.
  • F.2 Terminal Status of Hidden Intents: Terminal hidden-intent statuses distinguish intents completed directly, inferred through targeted questions, or provided by users without such questions.This intent-level distribution differs from the trajectory-level proactivity score averaged across tasks and repeated runs.
  • F.2 Terminal Status of Hidden Intents: Qwen3.6 Plus has the highest completed rate at 63.18, followed by Claude Opus 4.6 at 60.56 and GPT-5.4 at 56.84.These models also have the lowest provided rates, at 26.37, 28.20, and 30.87, respectively.
  • F.2 Terminal Status of Hidden Intents: Gemini 3.1 Pro, Kimi K2.5, and MiniMax M2.7 leave more hidden intents to users, with provided rates of 47.39, 45.85, and 43.81.The main variation comes from direct completion rather than targeted elicitation, whose rates remain low and tightly clustered across models.
  • F.3 Task-Type Breakdown of Performance: Research-centered workflows (A–C) and legal matter operations and handoffs (H) show relatively high COMP but lower PROC.Agents can often produce final deliverables once requirements are explicit, but are weaker at identifying latent goals, missing materials, and operational blockers earlier.
  • F.3 Task-Type Breakdown of Performance: Workflow structure shapes which capabilities are stressed, reinforcing that COMP and PROC capture different aspects of performance.Task-type diagnostics separate completion from proactivity across workflow structures.
  • F.3 Task-Type Breakdown of Performance: GPT-5.4 attains the strongest average PROC, while Claude Opus 4.6 achieves the strongest average COMP and Qwen3.6 Plus performs comparatively stably across both metrics.GPT-5.4 performs well on legal drafting (F), crisis communication (P), and administrative workflows (R); Claude remains competitive on research-oriented completion (A–C).

F.4 Failure Analysis · G Task Format · H Discussion

The failure analysis identifies recurring breakdowns in proactive intent resolution, context use, clarification, and artifact verification. The task format encodes explicit requests alongside latent requirements and evaluation criteria, illustrated through a survey-paper workflow.

  • F.4 Failure Analysis: The benchmark distinguishes checklist completion from proactive hidden-intent resolution when analyzing recurring failures.Checklist items measure task completion, while hidden intents measure whether requirements are resolved proactively rather than supplied by the user.
  • F.4 Failure Analysis: Agents may ignore recoverable information from earlier sessions, causing missed hidden intents or incorrect final artifacts.Examples include the Researcher dependency failure and a meal-planning contrast.
  • F.4 Failure Analysis: Agents often complete the visible request while leaving implicit preferences, formatting constraints, or task-specific requirements unresolved.High checklist completeness can still depend on user-provided hidden requirements.
  • F.4 Failure Analysis: Clarifications are effective when they target the specific missing requirement, whereas weaker trajectories require repeated user intervention.Targeted-followup successes provide positive contrasts in the cited figures.
  • F.4 Failure Analysis: Tool use can fail when agents do not verify that the produced artifact contains required content or matches the checklist.The Law Trainee trace uses SMS and Todoist but omits required handover details and reminder checks.
  • G Task Format: Each task is represented as a structured configuration containing the user-facing request, latent requirements, evaluation objectives, and metadata.Earlier tasks in the same group can provide opportunities to infer hidden intents, and task-specific details are anonymized.
  • G Task Format: The survey-paper example asks agents to organize relevant local papers by inferring a preferred topic focus, filtering candidates, and preparing a concise reading plan.Requirements include about [N] papers, technical summaries, official and code links when available, suitability judgments, and concrete follow-up actions.

H.1 Metric Relationship … I.1 Task Sources and Workflow Collection

π-BENCH separates proactivity from task completeness by distinguishing how requirements are surfaced from whether final outcomes satisfy them. Its tasks are built from sanitized, artifact-centric workflows across five roles, with realistic underspecification requiring agents to recover missing context through interaction.

  • H.1 Metric Relationship: Proactivity and completeness are distinct because final outcomes can improve through proactive requirement discovery or through users eventually stating those requirements.π-BENCH continues simulated interactions until every hidden intent is completed, elicited, or explicitly provided.
  • H.2 Hidden Intents and Checklists: Hidden intents encode unstated preferences, constraints, output conventions, and cross-session dependencies that agents should infer, ask about, or act on proactively.They evaluate how agents handle underspecified initial requests before users explicitly provide the information.
  • H.2 Hidden Intents and Checklists: Checklists define verifiable completion criteria for final trajectories or artifacts, including required files, formats, successful tool actions, and necessary content.A checklist item may depend on a hidden intent but is assessed only as an outcome requirement.
  • I.1 Task Sources and Workflow Collection: Tasks derive from realistic workflows for researchers, marketers, law trainees, pharmacists, and financiers, using domain-expert routines, deliverables, materials, and public task patterns.The sources span professional work and everyday knowledge work, including experiment analysis, paper writing, content planning, document review, and literature organization.
  • I.1 Task Sources and Workflow Collection: Task selection prioritizes artifact-centric workflows requiring interaction and realistic underspecification rather than tasks fully determined by the first user message.Correct completion must depend on constraints, preferences, prior decisions, or workspace context recovered through proactive behavior.
  • I Benchmark Construction and Annotation: Workflow materials are sanitized into synthetic but realistic instances by replacing private names, removing sensitive details, simplifying irrelevant background, and preserving access to required information.Required information remains available through the provided workspace, memory, or interaction protocol.

I.2 Task Construction Procedure … J.2 Checklist

The benchmark constructs realistic, underspecified tasks with hidden intents, checkable completion criteria, cross-session context, and validated workspaces. Case studies then show that agents differ in proactive intent resolution and checklist completeness, including targeted clarification successes and failures to preserve required outputs.

  • I.2 Task Construction Procedure: Annotators define realistic, checkable, well-scoped workflow goals that represent plausible user needs and support clear completion evidence.Goals avoid being too narrow to require interaction or too broad for reliable evaluation.
  • I.2 Task Construction Procedure: Task specifications separate explicit information from hidden intents, prepare workspace materials, and encode dependencies on earlier decisions, files, or preferences.Initial requests remain underspecified enough to test proactivity while concrete enough to support useful work.
  • I.3 Annotation Guidelines: Hidden intents encode latent constraints, preferences, output conventions, and prior-context dependencies, while checklist items define independently verifiable completion requirements.Separating them enables distinct measurement of proactive intent resolution and final task completion.
  • I.4 Quality Control: Validation binds tools, files, and workspace paths, tests reference workflows and expected artifacts, and uses pilot executions to expose setup or grading problems.Tasks are revised or removed when ambiguity or infeasibility is found.
  • I.4 Quality Control: Quality control checks leakage and recoverability so hidden intents remain unstated initially but can be recovered from sessions, artifacts, memory, tools, or clarification.The checks also detect missing files, invalid bindings, broken workspace paths, and infeasible setups.
  • J.1 Hidden Intents: In the Researcher failure case, Claude 4.6 Opus leaves several hidden intents merely provided, illustrating weak proactive resolution despite moderate final completeness.The trigger is an environment-generated paper recommendation feed rather than a direct user instruction.
  • J.1 Hidden Intents: In contrast, Researcher meal-planning traces expose only the one-week plan and RMB 20–30 constraint, while body profile, macro accounting, table structure, and Plan B remain hidden but recoverable.The passage compares DeepSeek V3.2 and Claude 4.6 Opus on the same task.
  • J.2 Checklist: Targeted follow-up succeeds when GPT-5.4 asks focused questions about missing crisis details, Kimi K2.5 asks how to send email when Gmail tools are unavailable, and GPT-5.4 consolidates Law Trainee handover requirements into one SMS.The contrasting Kimi K2.5 Law Trainee trace splits information across messages and omits the reply deadline and risk-flag instruction.

J.3 Cross-Session Dependency Design

Cross-session dependencies test whether agents can recover research themes, presentation preferences, and other hidden requirements from prior interactions. Claude 4.6 Opus carries forward prior context more effectively, while Kimi K2.5 repeatedly requests restatement and produces lower-quality task completion.

  • Researcher – Cross-Session Dependency: Claude 4.6 Opus carries over the user’s research theme and output conventions from earlier sessions, whereas Kimi K2.5 waits for repeated restatement.Prior context included a preference for a small paper batch, links, open-source status, and concrete follow-up paths.
  • Researcher – Cross-Session Dependency: Kimi K2.5’s failure to recover prior research context lengthens the conversation, causes drift from the annotated target set, and yields low proactivity with zero checklist completion.These failures occur despite several rounds of user-provided requirements.
  • Researcher – Claude 4.6 Opus: Claude 4.6 Opus uses memory to identify the research theme as think-with-image and begins filtering ICLR papers around image-driven reasoning and multimodal tool use.The trajectory marks intent I1 as completed.
Loading 2605.14678v3…