Source-linked AI summary
Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants
Deepak Nathani, Cheng Zhang, Chang Huan, Jiaming Shan, Yinfei Yang, Alkesh Patel, Zhe Gan, William Yang Wang, Michael Saxon, Xin Eric Wang
TL;DR
Proactive-assistant research lacks realistic user simulation because existing environments do not capture stateful, sequential interaction. Pare models apps with finite state machines and state-dependent user actions, then evaluates assistants on Pare-Bench’s 143 tasks; the best frontier models achieve only 42% success, while smaller models are bottlenecked by execution.
Problem
Prior proactive-assistant evaluations use passive, out-of-loop users, while realistic assessment requires users to interact and shape goal completion.
Method
Pare models user interaction with finite state machines and restricted state-dependent actions, while assistants retain flat API access; Pare-Bench evaluates 143 tasks.
Results
The best frontier models achieve only 42% success rate, and smaller models show execution as the primary bottleneck.
Takeaways & Limitations
Pare-Bench provides an interactive benchmark for studying goal inference, intervention timing, and multi-app orchestration in proactive assistants.
Takeaways & Limitations
The LM-based user simulator may miss human variation in fatigue, emotion, multitasking, personality, and trust.
Abstract
from arXiv · showhide
Proactive agents that anticipate user needs and autonomously execute tasks hold great promise as digital assistants, yet the lack of realistic user simulation frameworks hinders their development. Existing approaches model apps as flat tool-calling APIs, failing to capture the stateful and sequential nature of user interaction in digital environments and making realistic user simulation infeasible. We introduce Proactive Agent Research Environment (Pare), a framework for building and evaluating proactive agents in digital environments. Pare models applications as finite state machines with stateful navigation and state-dependent action space for the user simulator, enabling active user simulation. Building on this foundation, we present Pare-Bench, a benchmark of 143 diverse tasks spanning communication, productivity, scheduling, and lifestyle apps, designed to test context observation, goal inference, intervention timing, and multi-app orchestration.
1. Introduction
Pare addresses the difficulty of evaluating proactive assistants with realistic interacting users by modeling stateful app navigation and asymmetric access. It also introduces Pare-Bench, which evaluates seven assistants across 143 tasks and four application domains.
- Proactive assistants are needed because users may not know which actions require completion, making explicit requests cognitively demanding.
- Prior proactive-assistant evaluations rely on passive, out-of-loop users, preventing assessment of user-assistant goal completion through interaction.
- Pare creates ecologically valid user simulator agents that interact with proactive assistants for dynamic, scalable evaluation of goal-oriented impact.
- User simulators navigate app-specific and global finite state machines, selecting only functions enabled by the current interface state.
- Pare-Bench contains 143 diverse tasks across communication, productivity, scheduling, and lifestyle apps, testing context observation, goal inference, intervention timing, and multi-app orchestration.
- Pare uses asymmetric interfaces: users access state-dependent actions, while proactive agents access flat APIs for information gathering and task execution.
2. Related Work
Prior user-simulation benchmarks support tool-agent interaction but do not model mobile screen-by-screen navigation or fully interactive proactive assistance. Pare addresses these gaps with FSM-based mobile apps and asymmetric user-agent action spaces.
- Generative Agents, τ-bench, ToolSandbox, and τ2-bench established language-model user simulation for social, customer-service, conversational, and technical-support evaluation.
- Existing benchmarks do not target mobile environments or screen-by-screen phone-interface navigation.
- ProactiveAgent represents events textually without executable tool calls, so its gym cannot track environment state.
- ContextAgent evaluates proactive assistance on static sensory-data samples rather than through interactive simulation.
- Pare combines proactive goal inference with grounded execution in stateful mobile environments, where simulated users navigate FSM apps and agents access all-app APIs.
3. Proactive Agent Research Environment
Pare extends agent environments with active user simulation, modeling users as constrained actors who navigate stateful app interfaces while assistants access broader APIs. Its formal interaction loop captures asymmetric observations, proposal acceptance, environment events, and joint goal completion.
- Framework motivation: Pare extends ARE with active user simulator agents that interact with proactive assistants rather than remaining out of the loop.The framework models users and assistants as interacting agents in a shared environment.
- Formal interaction: Pare formalizes proactive assistance as Stackelberg interaction in which users act first and assistants infer goals before proposing plans.The agent’s plan depends on prior observations, both agents’ actions, current state, and environment events.
- Evaluation: The framework jointly evaluates goal success and plan acceptance after user and assistant actions conclude.A goal succeeds when its success indicator is true under the combined action sets and final environment state.
- Observation asymmetry: User actions are restricted by current state, and users receive limited observations that exclude the assistant’s actions, unlike the assistant’s broader information access.Pare routes user actions, assistant messages, and environment events to their intended recipients, with truncated notifications for users and full serialized content for assistants.
- Stateful interfaces: Users and assistants can affect the same underlying state through different interfaces: assistants call APIs directly, while users traverse app states and navigation commands.Each app FSM represents screens and transitions such as filling fields, navigating, and submitting requests.
- Stateful interfaces: Sending a message requires only two API calls in non-FSM frameworks but a sequential screen-navigation path in Pare.The Pare sequence opens Messages, searches for a conversation, opens it, and sends the message.
- Environment events: Pare scenarios combine initialized app state, event flows, and oracle validation, with scheduled events enabling dynamic situations over simulated time.Events can be scheduled absolutely or relative to other events and may broadcast updates to users and assistants.
4. User Simulator and Proactive Assistant
Pare separates proactive assistance into observation and execution while coordinating it with an active user simulator in a turn-based loop. The assistant proposes help from read-only observation, executes accepted plans across apps, and returns to observation after acceptance or rejection.
- System architecture: Pare orchestrates an active user simulator and proactive assistant through a Stackelberg loop in which the user acts before the assistant observes and acts.Both agents follow a Reasoning → Action → Observation cycle and may take multiple actions per turn.
- User Simulator: The user simulator receives tools dynamically from the current app and navigation state, and proposal acceptance or rejection tools appear only when a proposal awaits review.The simulator is instructed to reject vague, misunderstood, or unsupported proposals.
- Proactive Assistant: The proactive assistant consists of separate Observe and Execute sub-agents with observe, awaiting-confirmation, and execute modes.Awaiting confirmation is transient and lasts while the assistant waits for proposal acceptance.
- Proactive Assistant: In observe mode, the assistant monitors user actions and notifications using read-only tools, then either waits or sends a proposal to the user.Either control action ends the assistant’s turn.
- Proactive Assistant: Accepted proposals transition the assistant into execution mode with the proposed goal as its initial task.Rejected proposals return the assistant to observe mode without execution.
- Proactive Assistant: The executor accesses a flat API across all scenario apps, enabling autonomous multi-application task completion before returning to observation.It messages the user only on completion or when the task proves impossible.
5. Pare-Bench
Pare-Bench is a 143-scenario benchmark for evaluating proactive agents across diverse applications and contexts. Its scenarios are generated through an LLM-based pipeline and can include failures and noise to test signal discrimination.
- Benchmark: 143 scenarios evaluate proactive agents across diverse contexts and applications.The benchmark is built on the Pare framework.
- Scenario generation: An LLM-based scenario-generation agent produces candidate scenarios through story generation, app-data population, event-flow construction, and validation.The pipeline addresses the limited scalability of manually authored scenarios.
- Scenario coverage: All scenarios include core communication and navigation applications, while application usage varies across the benchmark.The distribution of applications is shown in the benchmark’s application-use overview.
- Robustness: Simulated tool failures and spurious notifications vary noise density to evaluate whether assistants distinguish signal from noise.Noise events are scheduled with a Poisson process and interleaved with scenario events.
6. Experiments
Pare-Bench evaluates seven proactive assistants across repeated runs, proposal behavior, information gathering, execution, tool failures, and environmental noise. Frontier models lead overall, while smaller models show weaker consistency and execution, with robustness varying by model.
- Evaluation Setup: Seven language models are evaluated on Pare-Bench across four runs using acceptance, execution success, proposal rate, and information-gathering metrics.Success is reported through Success@k, Success^k, and Success Rate to capture one-run success, reliability, and average performance.
- Overall Results: 42.1% and 42.0% success rates place Gemini 3 Flash and Claude 4.5 Sonnet comparably at the top, while Gemma 3 4B Instruct reaches only 3.0%.Qwen 3 4B Instruct leads the open-weights models across all reported metrics.
- Consistency: 17.0× versus 3.3× reductions from Success@4 to Success^4 show that Llama is less consistent across repeated runs than Claude.Llama falls from 23.8% to 1.4%, while Claude falls from 60.8% to 18.2%.
- Proposal Quality and Efficiency: Claude combines the lowest proposal rate, 12.8%, with the highest acceptance rate, 78.2%, whereas GPT-5 proposes most often at 28.1%.Gemini 3 Flash attains similar success to Claude but uses more proposals and has lower acceptance; Gemma is both passive and inaccurate when proposing.
- Information Gathering and Execution: Approximately 20% more read actions by top-performing models than Qwen and Llama, and nearly twice as many as Gemma, correlate with higher overall performance.Qwen’s 63.7% acceptance but 18.5% success suggests execution is its primary bottleneck.
- Robustness: 40% tool failure leaves Claude near 40–45% success, while Llama drops from 20% to 8%; under noise, Claude and Qwen remain stable but Gemini 3 Flash, GPT-5, and Llama degrade.Proposal and acceptance rates remain stable across the tested failure and noise levels.
7. Conclusion
The paper concludes with Pare, Pare-Bench, and the Observe-Execute architecture for evaluating proactive assistants through active user simulation. Experiments report only 42% success for the best frontier models and identify execution as a primary bottleneck for smaller models.
- Contributions: Pare combines active user simulation with FSM-based stateful app interfaces, while Pare-Bench contains 143 tasks testing goal inference, intervention timing, and multi-app orchestration.The paper also introduces the Observe-Execute agent architecture.
- Findings: 42% success is achieved by the best frontier models, while execution is identified as the primary bottleneck for smaller models.The Observe-Execute design can partially address this bottleneck.
Impact Statement
The impact statement emphasizes privacy and user autonomy in proactive assistants. It advocates edge deployment, API-level observation, permission before execution, and releasing Pare and Pare-Bench for reproducible research.
- Privacy: Edge deployment is proposed to avoid uploading continuously observed user data to external servers.The paper links this preference to the privacy implications of proactive assistants observing user actions.
- Privacy: API-level abstraction creates a privacy boundary in which agents observe actions rather than everything visible on screen.This describes the paper’s chosen representation of user actions.
- User Autonomy: The Observe-then-Execute architecture preserves user control by requiring permission before task execution.The impact statement frames human control as necessary for valuable proactive assistants.
- Research Impact: Releasing Pare and Pare-Bench is intended to enable reproducible research and support development of capable, privacy-preserving proactive assistants.The stated scope is community research and development.
A. Limitations and Future Work
Pare’s evaluation scope is bounded by API-level app interactions, LM-based user simulation, a symmetric observe-execute setup, and 143 scenarios that omit some long-horizon and world-knowledge tasks.
- Scope boundaries: Pare models app interactions as tool calls rather than visual screen interactions, so it does not test visual grounding for multimodal deployment.The authors describe this as a deliberate choice supporting a privacy boundary and scalable evaluation.
- Scope boundaries: LM-based simulated users may omit human differences in fatigue, emotion, multitasking, personality, and trust.The authors report preserved relative rankings across user models but leave personalization through theory-of-mind approaches open.
- Scope boundaries: The main experiments use one model for both observation and execution rather than the intended asymmetric deployment with smaller observation and larger execution models.The asymmetric configuration remains unevaluated, despite motivating future work on pairing on-device observation with larger execution.
- Future work: Pare-Bench’s 143 scenarios may omit long-horizon, multi-day tasks and tasks requiring world knowledge beyond its app ecosystem.The authors propose expanding the app ecosystem and inter-app dependencies, including safety-critical banking actions.
- Future work: Future extensions include reinforcement-learning training with acceptance and success rewards and multimodal evaluation using UI screenshots alongside APIs.The paper frames these as possible ways to connect Pare’s current abstraction to broader agent training and deployment settings.
B. Proposal Decision Analysis
The proposal analysis separates direct acceptance, rejection, and continued context gathering, revealing differences in intervention timing and how gathered context resolves across models.
- Ternary decisions: Claude has the highest direct acceptance rate at 72.1% ± 1.4% and the lowest rejection rate at 7.8% ± 0.8%.The table reports ternary proposal decisions across models with standard error over four runs.
- Ternary decisions: 74.7% ± 2.6% of Gemma’s proposals trigger context gathering, while only 16.0% are directly accepted.This pattern is interpreted in the passage as indicating that many Gemma interventions are premature.
- Ternary decisions: Qwen and Llama trigger context gathering more often than frontier models, at 26.5% and 29.1%, respectively.The passage associates these higher rates with proposals that are often premature.
- Context resolution: Across models, 62–80% of context-gathering proposals are truncated under the experiment’s 10-turn limit.The authors identify the turn budget as the reason for truncation and suggest studying higher budgets.
- Context resolution: Context gathering converts to acceptance at 25–30% for frontier models but only 10.1% for Gemma.Conversion to rejection is also higher for smaller models, at 9–10%, than for Claude and GPT-5, at 3.7–4.6%.
C. User Simulator Ablation
The user-model ablation varies simulated users while holding proactive assistants fixed, finding large shifts in absolute metrics but stable relative assistant rankings.
- User-model effects: Claude users are strictest, producing 23–49% acceptance and 13–26% success across proactive assistants.Qwen users are most permissive, with 74–80% acceptance but success still ranging from 11–44%.
- User-model effects: GPT-5-mini provides intermediate acceptance and success spread, supporting its use as the default user model in the main experiments.The passage states that this spread helps discriminate between proactive models.
- Ranking stability: Relative proactive-assistant ordering is preserved across all three user models despite large shifts in absolute results.Claude has the highest acceptance regardless of the user model, while Claude proposes least and GPT-5 most.
- Observation behavior: Qwen users increase observed read actions to 20–26 versus 15–20 for other user models, showing that simulated user behavior affects exploration.Even with high read actions and a permissive user, Qwen as proactive assistant achieves 0% Pass^4.
- Environment design: Pare formalizes interactions as a Stackelberg POMDP in which the user acts first and the proactive agent observes before deciding whether to intervene.The environment includes state-dependent user actions, state-independent agent actions, deterministic transitions with optional stochastic tool failures, and dual acceptance/success rewards.
E.1 Scenario Generation Agent
Pare generates benchmark scenarios through a four-stage pipeline that creates descriptions, app states, event flows, and validation functions, then checks execution and semantic coherence. The resulting scenarios model diverse stateful applications and user workflows.
- Scenario abstraction: Each scenario comprises initial app states, an environment event flow, and validation.These components define the scenario abstraction used by the generation pipeline.
- Generation pipeline: A code-generation agent creates scenario descriptions, populates initial app data, builds event flows with oracle actions, and generates validation conditions.The agent receives the Pare codebase read-only and a selection of target apps.
- Quality control: Scenario descriptions are checked for uniqueness with LLM-as-a-judge, while generated functions are executed in oracle mode to catch syntax and execution failures.Failed checks provide execution feedback for retry, and a human verifies semantic coherence.
- Application coverage: Pare-Bench distributes scenarios across applications including cab, notes, email, calendar, contacts, reminders, shopping, and messaging.The benchmark always includes the core PASAgentUserInterface and System applications, which support assistant-user communication and inter-app navigation.
- Stateful interaction: Stateful app workflows expose navigation-dependent actions, such as composing email through sequential transitions from starting composition to recipients, attachments, and sending.The application descriptions also specify distinct states and state-dependent operations for notes, calendar, contacts, reminders, shopping, and messaging.
G. Scenario Example
The example shows a proactive assistant responding to an incoming email by inferring a changed apartment-search constraint, proposing corrective actions, and waiting for confirmation before altering saved data. The environment supports this interaction through state-based apps, notifications, and strict user-agent decision rules.
- Scenario Example: An external email about an apartment-search budget is delivered through the event loop to both user-facing and assistant-facing contexts.The user sees a truncated notification and can inspect the email, while the assistant observes the complete message.
- Scenario Example: The assistant retrieves saved apartments, evaluates them against the new budget constraint, and identifies listings that exceed the monthly limit.It summarizes the violations and generates a proposal rather than immediately changing the saved list.
- Scenario Example: Only explicit user approval triggers removal of over-budget apartments, producing validated state transitions that leave budget-compliant options saved.The assistant waits for confirmation before executing the state-altering actions.
- User-agent decision rules: The simulator evaluates proposals against intent, recent context, available apps, and helpfulness, accepting only when it is extremely certain and otherwise rejecting.It is instructed not to accept every proposal and to reject vague, unhelpful, misunderstood, or unsupported requests.
- Environment interaction: The user simulator interacts with one active app at a time, while available actions vary with the current app state and environment changes follow user actions.System navigation preserves previously opened app states and supports switching among already-open applications.
- Proactive assistant: The proactive assistant observes user actions and notifications, infers goals, proposes tasks when confidence is sufficient, and remains silent when uncertain.Its observations include both user behavior and environment events such as incoming emails and calendar reminders.