PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents

Yuxiang Chai; Shunye Tang; Han Xiao; Rui Liu; Hongsheng Li

Source-linked AI summary

PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents

Yuxiang Chai, Shunye Tang, Han Xiao, Rui Liu, Hongsheng Li

arXiv:2603.08013v1Mar 9, 2026cs.AI

TL;DR

Existing GUI agents depend on explicit, detailed instructions, whereas proactive assistants should infer latent user goals from continuous visual streams despite multitasking and noise. The paper introduces PIRA-Bench for this evaluation and PIRF, a memory-and-reflection baseline for tracking task threads. The evaluation finds high recall but substantial over-proactivity and hallucinated intents under visual noise, with consistent gains from PIRF.

Problem
Reactive GUI agents require explicit detailed prompts, but real-world screen activity contains interleaved tasks, distractions, and noisy segments that proactive assistants must interpret.
Method
The paper introduces PIRA-Bench and PIRF, which uses dynamic memory and reflection to evaluate and support proactive intent recommendation from visual trajectories.
Results
Frontier models show high recall but significant over-proactivity and hallucinated intents under visual noise, while PIRF produces consistent gains.
Takeaways & Limitations
PIRA-Bench provides a systematic setting for assessing latent-intent discovery, interleaved-task disentanglement, and personalized recommendations in GUI assistants.
Takeaways & Limitations
The benchmark models real-world trajectories as disjointed unions of task-specific sub-trajectories alongside irrelevant noise.

Abstract

from arXiv · show

Current Graphical User Interface (GUI) agents operate primarily under a reactive paradigm: a user must provide an explicit instruction for the agent to execute a task. However, an intelligent AI assistant should be proactive, which is capable of anticipating user intentions directly from continuous visual inputs, such as mobile or desktop screenshots, and offering timely recommendations without explicit user prompting. Transitioning to this proactive paradigm presents significant challenges. Real-world screen activity is rarely linear; it consists of long-horizon trajectories fraught with noisy browsing, meaningless actions, and multithreaded task-switching. To address this gap, we introduce PIRA-Bench (Proactive Intent Recommendation Agent Benchmark), a novel benchmark for evaluating multimodal large language models (MLLMs) on continuous, weakly-supervised visual inputs. Unlike reactive datasets, PIRA-Bench features complex trajectories with multiple interleaved intents and noisy segments with various user profile contexts, challenging agents to detect actionable events while fitting to user preferences. Furthermore, we propose the PIRF baseline, a memory-aware, state-tracking framework that empowers general MLLMs to manage multiple task threads and handle misleading visual inputs. PIRA-Bench serves as an initial step toward robust and proactive GUI-based personal assistants.

1 Introduction

The paper shifts GUI agents from reactive execution toward proactive intent recommendation from continuous visual context. It introduces PIRA-Bench to evaluate this setting and PIRF to track multitasking states while reducing hallucinations in noisy trajectories.

Motivation: Current GUI agents require detailed human instructions, burdening users and risking failure when contextual details are forgotten or omitted.Reactive agents may instead require tedious, step-by-step clarification.
Proactive Intent Recommendation: PIR agents continuously analyze GUI screenshot streams to infer latent goals and recommend actionable instructions before users formulate commands.Examples include booking a restaurant, setting a reminder, and adding a calendar event.
PIRA-Bench: PIRA-Bench evaluates proactive recommendation using 100 annotated trajectories averaging 32 sequential screenshots and three user profiles.Its scenarios cover direct recommendation, profile-dependent prediction, and noise rejection, including interleaved intents and distracting segments.
PIRA-Bench: The benchmark tests whether agents can disentangle concurrent activities, use user profiles, and avoid false predictions when no actionable intent exists.These requirements reflect multitasking and intentional visual noise in the trajectories.
PIRF: PIRF equips general MLLMs with iterative visual processing, dynamic memory, and reflection-based auto-deletion for tracking crossed intents and mitigating hallucinations.The memory module records ongoing task and profile states, while reflection removes outdated or completed intents.

2 Related Work

Related work situates GUI agents within broader progress toward automated digital interaction and proactive personal agents. It contrasts specialized GUI navigation with continuous, cross-environment automation initiatives.

GUI Agents: GUI agents use visual understanding to execute complex action sequences across mobile and desktop operating systems.Prior work has emphasized instruction-following and visual grounding for GUI navigation.
Proactive Personal Agents: Broader AI research is moving toward autonomous personal agents that continuously connect digital environments with general task execution.Open-source initiatives integrate language models with local operating systems and cross-platform messaging APIs.

3 Proactive Intent Recommendation

The paper formulates proactive intent recommendation as predicting actionable future goals from passive screenshot trajectories and user profiles rather than explicit instructions. It also models real-world activity as interleaved task segments mixed with irrelevant noise.

3.1 Reactive GUI Agent Paradigm: Standard GUI agents receive an explicit instruction, visual state, and action space, then generate actions through a policy over the observed history.Without human prompting, the agent cannot determine the goal state and remains idle.
3.2 Proactive Intent Recommendation: PIR maps a sequence of passively captured GUI screenshots and a user profile to a set of likely future actionable intents.Each intent may be expressed as a natural-language instruction or structured command.
3.3 Real-World Complexity: Real-world trajectories combine non-contiguous sub-trajectories for multiple concurrent tasks with irrelevant noise from app switching, browsing, or idle scrolling.This structure reflects multitasking and distraction rather than a single linear task.
3.3 Real-World Complexity: A successful PIR agent must assign temporal credit, disentangle interleaved state subsets, and suppress intent generation from noise.The formulation requires mapping task-specific frame subsets to their corresponding distinct intents.

4 PIRA-Bench

PIRA-Bench is a unified benchmark for proactive intent recommendation from passive GUI streams, combining realistic multitasking, personalization, and noise. It uses consensus annotations and semantic judging to assess intent accuracy and hallucination restraint.

Dataset construction: PIRA-Bench contains 100 passive GUI trajectories from mobile and desktop environments, presented as a unified mixed distribution of realistic challenges.The dataset is designed to reflect unpredictable device usage rather than isolated capability subsets.
Dataset construction: Every trajectory includes noise frames and three distinct user profiles, enabling evaluation of distraction filtering and scalable personalization.Profiles vary in socio-economic status, preferences, and characteristics; noise includes irrelevant app switching, idle screens, and random browsing.
Evaluation scenarios: The benchmark covers direct recommendation, profile-dependent prediction, and pure-noise rejection within one holistic testbed.These scenarios jointly evaluate predictive accuracy, personalization capability, and operational restraint.
Ground-truth annotation: Three independent annotators label actionable intents for each trajectory and profile, with consensus producing an empty ground truth for pure-noise cases.The annotation process addresses subjectivity in latent goals and profile constraints.
Evaluation pipeline: An LLM judge semantically compares predicted and ground-truth intent sets while checking compliance with user-profile constraints.Gemini-3-flash performs the comparison instead of relying on exact lexical matching.
Metrics: PIRA-Bench reports average intent F1 on positive trajectories, normalized false-positive performance on negative trajectories, and a multiplicative final reliability score.The final score combines proactive capability with a hallucination penalty, so false positives affect overall reliability.

5 Proactive Intent Recommendation Framework (PIRF)

PIRF wraps a general MLLM in a sequential, memory-aware architecture for disentangling continuous, interleaved GUI intents. It tracks state transitions frame by frame and uses reflection to remove stale intents from memory.

Architecture: PIRF addresses long-horizon context retention and hallucination susceptibility by adding structured state tracking and reflection around a general MLLM.The framework is designed specifically for continuous intent disentanglement in noisy visual streams.
Architecture: PIRF processes visual streams sequentially rather than consuming an entire trajectory at once, supporting continuous intent updating.This design avoids the computational expense and real-world inapplicability of processing the full trajectory simultaneously.
Dynamic memory: Its dynamic Memory Module anchors user profiles and tracks ongoing multitasking states across interleaved tasks.The memory design supports maintaining multiple task threads while processing new observations.
Intent action space: At each frame, PIRF compares the current visual observation with suspended intents and emits a structured state transition.The framework’s action space includes CREATE, RESUME, UPDATE, and IDLE operations.
Reflection and deletion: A continuous reflection and auto-deletion mechanism removes stale intents independently of primary state transitions.The mechanism targets memory bloat caused by abandoned tasks and changing context.

6 Experiments

PIRA-Bench evaluates MLLMs and PIRF on intent detection and noise robustness, comparing naive context processing, structured memory and reflection, and human performance. Results show that visual noise exposes over-proactivity, while PIRF improves restraint across models but remains far below human performance.

Main Results: GPT-5.2’s naive setting reaches 83.37% Recall but only 31.95% Precision and 31.31% FPSnorm, yielding a 12.76 Sfinal.Its high sensitivity is accompanied by frequent hallucinated intents from noise frames.
Main Results: PIRF consistently outperforms the Naive baseline across evaluated models by combining dynamic memory with reflection.For GPT-5.2, PIRF raises Precision to 50.52% and FPSnorm to 43.90%, while Recall reaches 84.54%.
Main Results: Seed-1.8 with PIRF achieves the highest reported Sfinal of 28.05, driven by 50.36% FPSnorm despite 72.67% Recall.The result reflects a conservative balance that favors avoiding hallucinations during noise.
Main Results: Human performance reaches 90.35 Sfinal, compared with 28.05 for the best model; humans also achieve 98.76% Precision and 96.23% noise robustness.GPT-5.2’s 84.54% Recall approaches the human 89.67% benchmark, but humans require approximately 15 to 20 times more inference time.
Ablation Study: GPT-5.2 Precision falls from 92.23% on Clean trajectories to 50.52% on Noised trajectories, while Gemini-3.1-Pro drops from 85.28% to 53.05%.Clean trajectories retain intent-relevant frames, whereas Noised trajectories include injected irrelevant frames.
Ablation Study: Irrelevant frames such as idle scrolling and random browsing can be misinterpreted as meaningful triggers, causing models to hallucinate nonexistent intents.This distraction pattern limits reliable operation in unconstrained visual environments.
Ablation Study: GPT-5.2 Recall increases from 83.57% to 84.54% when noise is added, indicating that clutter can trigger more aggressive intent predictions.The recall gain comes alongside the precision degradation reported for noisy trajectories.

7 Conclusion

PIRA-Bench introduces a benchmark for proactive GUI assistants using diverse trajectories, user profiles, and visual noise, while PIRF combines dynamic memory with reflection. The evaluation finds high recall but substantial over-proactivity in frontier models, whereas PIRF provides consistent gains and supports more discerning action selection.

Conclusion: PIRA-Bench evaluates latent-intent discovery, interleaved-task disentanglement, and personalized recommendations across 100 trajectories with distinct user profiles and realistic visual noise.It is designed as a systematic assessment of GUI-based proactive assistants.
Conclusion: PIRF integrates dynamic memory and reflection to address the multitasking and visual-noise challenges of proactive intent recommendation.The framework is proposed as a baseline architecture for these settings.
Conclusion: Frontier models show high recall but significant over-proactivity and hallucinate intents when visual noise is present.The conclusion identifies restraint under noise as a central unresolved challenge.
Conclusion: PIRF’s consistent gains indicate that structured state tracking and self-reflection are viable pathways for mitigating these failures.The paper positions these mechanisms as a foundation for future proactive-agent research.