Source-linked AI summary
ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data
Yuanbo Tang, Huaze Tang, Tingyu Cao, Lam Nguyen, Anping Zhang, Xinwen Cao, Chunkang Liu, Wenbo Ding, Yang Li
TL;DR
Existing proactive-agent datasets often rely on synthetic interactions or isolated tasks, omitting authentic decision patterns and the behavioral context preceding assistance. ProAgentBench addresses this gap with a privacy-compliant benchmark of continuous real-world sessions and a hierarchical timing-and-content framework, finding that context, long-term memory, and real-world training data improve prediction performance.
Problem
Existing datasets lack authentic real-world interaction patterns and continuous pre-assistance context because they rely on synthesized data or isolated tasks.
Method
ProAgentBench combines privacy-compliant continuous workflow collection with a hierarchical framework that separates intervention timing prediction from assistance-content generation.
Results
Context and long-term memory significantly enhance prediction accuracy, while real-world training data substantially outperforms synthetic alternatives.
Takeaways & Limitations
The benchmark provides a reusable evaluation setting grounded in continuous working scenarios, authentic bursty interactions, and pre-assistance behavioral context.
Takeaways & Limitations
The dataset may generalize imperfectly because participants come from limited professions, technology stacks, regions, and languages, while sampling and privacy filtering can omit interactions.
Abstract
from arXiv · showhide
Proactive agents that anticipate user intentions without explicit prompts represent a significant evolution in human-AI interaction, promising to reduce cognitive load and streamline workflows. However, existing datasets suffer from two critical deficiencies: (1) reliance on LLM-synthesized data that fails to capture authentic human decision-making patterns, and (2) focus on isolated tasks rather than continuous workflows, missing the pre-assistance behavioral context essential for learning proactive intervention signals. To address these gaps, we introduce ProAgentBench, a rigorous benchmark for proactive agents in working scenarios. Our contributions include: (1) a hierarchical task framework that decomposes proactive assistance into timing prediction and assist content generation; (2) a privacy-compliant dataset with 28,000+ events from 500+ hours of real user sessions, preserving bursty interaction patterns (burstiness B=0.787) absent in synthetic data; and (3) extensive experiments that evaluates LLM- and VLM-based baselines. Numerically, we showed that long-term memory and historical context significantly enhance prediction accuracy, while real-world training data substantially outperforms synthetic alternatives. We release our dataset and code at https://anonymous.4open.science/r/ProAgentBench-6BC0.
1. Introduction
ProAgentBench addresses limitations in proactive-agent datasets by introducing a benchmark built from authentic, continuous user workflows. It combines privacy-compliant real-world data with a hierarchical framework for predicting when to intervene and how to assist.
- Proactive agents infer user intentions from context without explicit prompts and aim to provide timely assistance that reduces cognitive load.
- Existing datasets rely heavily on synthesized interactions and isolated tasks, missing authentic decision-making, bursty workflows, and pre-assistance behavioral context.
- ProAgentBench collects over 28,000 events from 500+ hours of continuous sessions using privacy-compliant collection, preserving burstiness B = 0.787 and pre-assistance context.
- Its hierarchical framework separates proactive assistance into predicting when to intervene and generating contextually appropriate assistance content.
- Experiments show that context and long-term memory improve prediction accuracy, while real-world training data substantially outperforms synthetic data.
2. Related Work
Prior work covers proactive assistance, device-based context awareness, mobile interaction, and screen-based task execution, but these resources generally do not capture continuous organic workflows before assistance. ProAgentBench targets this missing setting with privacy-preserving, temporally dense workflow data.
- Prior proactive-agent benchmarks include real-world events, wearable context, and long-term mobile interaction data across several application domains.
- Screen-interaction datasets provide mobile screenshots, web tasks, and functional environments, but primarily target execution of predefined goals.
- These datasets lack antecedent behavioral context, temporal density, and privacy-preserving methods needed for real PC work scenarios.
- ProAgentBench addresses the gap by recording continuous workflows containing both pre-assistance behavior and subsequent interaction events.
3. Problem Definition and Formulation
The benchmark models proactive assistance from temporal screen snapshots, timestamps, application metadata, historical information, and user profiles. Its hierarchical pipeline first predicts whether intervention is needed, then generates assistance content when intervention is warranted.
- Problem Definition and Formulation: The proactive agent continuously monitors current screen snapshots and initiates contact when it detects a need for service.
- Problem Definition and Formulation: The system models behavioral patterns and current contextual background to determine whether assistance is required and infer user intent.
- Temporal Snapshot Sequence Inputs: Each temporal snapshot contains a screen image, timestamp, and application metadata, forming the historical observation sequence O1:t.
- Temporal Snapshot Sequence Inputs: User information includes historical interactions and long-term behavioral patterns alongside profile attributes such as occupation, expertise, and preferences.
- A Hierarchical Pipeline for Proactive Assistance: The When to Assist stage monitors activities and predicts the optimal intervention moment, avoiding premature interruptions and missed opportunities.
- A Hierarchical Pipeline for Proactive Assistance: When to Assist is evaluated with accuracy, precision, recall, and F1 Score to capture reliability, trigger quality, missed assistance, and interruption trade-offs.
- A Hierarchical Pipeline for Proactive Assistance: When Bt = 1, the How to Assist stage generates natural-language content Ct in the text space V.
4. Dataset Overview
ProAgentBench combines real-user workflow logs with long-term temporal analysis to characterize proactive-assistance data. The dataset exhibits heavy-tailed, bursty human interactions that synthetic data does not reproduce.
- Dataset motivation: ProAgentBench compares real-user, long-term interaction data with existing proactive-assistance benchmarks that often rely on synthetic or simulated settings.The comparison targets authentic temporal dynamics and long-term user context.
- Dataset composition: 28,528 total events include 7,222 LLM-related events, covering information retrieval, knowledge Q&A, data analysis, code programming, and content generation.LLM-related events comprise approximately 25.3% of the dataset.
- Dataset composition: The dataset’s LLM-related activity spans browsers, file management tools, IDEs, and office software, with DeepSeek and Gemini among the leading identifiable providers.Web browsers account for 62.53% of LLM-related events.
- Temporal statistics: Human interaction inter-event times are heavy-tailed and fit a power law with exponent α = 1.50, supported over an exponential alternative.The log-likelihood ratio is 2951.48 with p = 7.83 × 10^-100.
- Temporal statistics: Human records have burstiness B = 0.787, whereas synthetic data has B = 0.166 and is better fit by an exponential model.The synthetic process therefore does not reproduce the bursty timing observed in human behavior.
5. Data Collection, Privacy Protection, and Automatic Annotation
The data pipeline collects continuous computer-use traces, protects sensitive content through layered review and filtering, and annotates LLM events using multimodal event windows. This design combines data quality, privacy protection, and contextual annotation.
- Data collection: LifeTrace1 collects screenshots at 1Hz and synchronized application logs, then segments continuous activity into events based on application switching.Invalid events are filtered and duplicate records are removed using hashing.
- Privacy protection: Privacy protection combines VLM screening, volunteer review with final retention control, and rule-based filtering, with high-risk data permanently deleted.The three-stage process combines automated detection and human oversight.
- Automatic annotation: The annotation workflow aggregates image sequences, OCR text, and metadata within event windows rather than analyzing screenshots independently.Qwen3-VL-Plus classifies platforms and interaction types and extracts conversation history.
- Evaluation setup: Table 2 evaluates prompt-based methods across both timing prediction and content prediction using accuracy, precision, recall, F1, intention accuracy, and semantic similarity.The table compares results across different models and identifies the best and second-best values in each column.
6. Experiments and Results
Experiments evaluate proactive assistance across models, prompt strategies, historical context lengths, memory methods, and training-data sources. Results show that context, long-term memory, and real-world training improve performance, while content generation remains difficult.
- Base results: Deepseek-V3.2 reaches 64.4% accuracy on When to Assist, while Qwen3-VL-Plus reaches 37.1% intention accuracy on How to Assist.Semantic similarity remains low across models, ranging from 0.275 to 0.286.
- Prompting methods: CoT prompting produces mixed results: it helps larger models but can harm smaller open-source models on implicit pattern-recognition tasks.The effect depends on model capacity and behavioral tendencies.
- Prompting methods: Self-Consistency improves Llama-3.1-8B-Instruct accuracy from 57.3% to 58.8% and Qwen3-VL-8B-Instruct accuracy from 51.7% to 52.9%.It does not significantly improve intention accuracy or semantic similarity.
- Overall findings: The benchmark remains challenging because timing prediction is easier than intention prediction, advanced prompting may hurt performance, and semantic similarity remains low.These observations indicate substantial room for more contextually appropriate assistance generation.
- Historical context: Longer historical context improves both timing F1 and intention accuracy, with intention gains becoming marginal beyond five minutes.The experiment varies historical windows from 10 seconds to 10 minutes.
- Long-term memory: Knowledge Graph memory increases overall Accuracy by 11.8%, from 0.537 to 0.601, and Intention Accuracy by 26.9%, from 0.312 to 0.396.Among the compared memory strategies, Knowledge Graph produces the largest improvement over Zero-shot.
- Training data: Fine-tuning LLaMA-3.1-8B-Instruct on real-world data raises Accuracy from 57.3% to 74.0%, while real-world training consistently outperforms synthetic training across metrics.The comparison directly tests the value of authentic user data for proactive assistance.
7. Conclusion
ProAgentBench evaluates proactive agents in continuous real-world workflows using privacy-compliant data and a hierarchical timing-and-content framework. Its experiments identify real-world training data and long-term memory as pivotal for performance.
- Conclusion: ProAgentBench captures over 28,000 events from 500+ hours of authentic activity while preserving pre-assistance behavioral patterns.The benchmark is designed for proactive agents operating within continuous real-world workflows.
- Conclusion: The benchmark decomposes proactive assistance into timing prediction and content generation and finds that real-world data and long-term memory integration are pivotal for agent performance.The authors present the benchmark as a foundation for context-aware proactive AI systems.
8. Impact Statements
The paper acknowledges dataset, privacy, and broader-impact limitations, including restricted participant diversity, sampling and annotation constraints, surveillance risks, and possible misuse.
- Limitations: Participant bias may limit generalization across professions, technology stacks, regions, languages, operating systems, and application ecosystems.The 1Hz sampling rate may miss very short interactions, while unstable window titles can introduce annotation errors.
- Limitations: Aggressive privacy filtering may exclude sensitive-content interactions and bias the dataset toward less privacy-sensitive workflows.
- Ethics Statement: Screen-level behavioral data carries inherent surveillance risks if misused, despite access controls, de-identification, and research-only licensing.
- Broader Impact: Proactive assistants may improve productivity and reduce cognitive load, but bypassed safeguards could enable intrusive applications or over-reliance on AI.
- Future Work: Future work includes richer sensor modalities, stronger long-range sequence models, and online user studies of proactive assistants.
A. Data Release & Usage
The release provides structured, auditable dataset artifacts linking screenshots, events, annotations, and curation decisions, with intention distributions documenting usage scenarios.
- Release and Access: The public release includes de-identified screenshots, derived features, event-level summaries, and evaluation protocols, while raw screenshots remain access-controlled.
- Release and Access: A research-only license prohibits re-identification and commercial applications, alongside privacy-minimizing guidelines and reproducible baseline scripts.
- Data Structure: The SQLite database stores core screenshot and event logs, while CSV/JSON/JSONL files record traceable labeling and filtering decisions.
- Data Structure: Screenshots and events are linked by event identifiers, enabling timeline alignment, event reconstruction, and screenshot-to-event traceability.
- Data Organization: Participant directories organize screenshots, databases, and semantic annotations by participant and event keys for reproducible alignment.
- Intention Distribution: Information lookup and knowledge Q&A together account for over 55% of LLM events, while remaining intentions form a long-tail distribution.
B. Data Collection, Privacy Protection, and Automatic Annotation
The pipeline collects continuous computer-use data, protects privacy through automated and human review, annotates events with multimodal temporal context, and augments models with historical memory and persona summaries.
- Data Collection: The dataset collection combines participant consent, withdrawal controls, multi-stage privacy protection, and event-level annotation across professional scenarios.
- Data Collection: LifeTrace collects 1Hz screenshots and application logs, then segments continuous activity into discrete application-window events.
- Annotation and Quality Control: Quality filtering removes events shorter than 3 seconds, reviews events longer than 1 hour, requires valid screenshots, and prioritizes sufficiently documented LLM events.
- Annotation and Quality Control: 97.6% annotation success was achieved through retries, parsing recovery, default VLM annotations, deduplication, and manual inspection.
- Privacy Protection: Privacy protection combines VLM detection, volunteer decisions to retain, blur, or delete, and deterministic rule-based validation.
- Annotation and Quality Control: Event-level annotation samples up to six screenshots, combines OCR and metadata, and uses temporal context rather than isolated screenshot classification.
- Memory Augmentation: Retrieval-augmented memory retrieves up to five temporally valid historical neighbors and injects them as narrative memory blocks into prompts.
- Persona Memory: Prototypical-diverse sampling balances cluster-central events with heterogeneous behavioral coverage when constructing persona summaries.
E.2. Context Time Window Length Ablation
Longer historical context generally improves both timing and intention prediction, but gains diminish beyond five minutes, making that window an efficiency-performance compromise.
- Ablation Results: Longer context windows generally improve Accuracy, Precision, Recall, and F1 Score for When to Assist, plus Intention Accuracy for How to Assist.
- Ablation Results: Performance gains show diminishing returns beyond the 5-minute mark, suggesting five minutes balances behavioral coverage and computational efficiency.
- Evaluation Setup: Figure 8 reports results across time windows from 30s to 10m for both When to Assist and How to Assist.
- Related Result: Real-world training data consistently outperforms synthetic data across all reported metrics and fine-tuning strategies.
F.1. Ablations 1: Impact of Agent Reasoning Strategies
The ablations compare prompting strategies, input modalities, and latency. Self-Consistency is the most reliable strategy, while text-only inputs often match or outperform multimodal inputs and CoT incurs substantial latency.
- Reasoning strategies: Self-Consistency maintains robust performance where CoT can fail, including Qwen3-VL-8B-Instruct text-only F1 scores of 66.7% versus 22.4%.This slightly exceeds the Zero-shot baseline of 66.1%.
- Reasoning strategies: CoT has model-dependent effects, improving larger models but harming smaller open-source models on implicit pattern-recognition tasks.Its reasoning can shift models toward aggressive triggering or excessive conservatism.
- Input modalities: Multimodal inputs can reduce performance; Qwen3-VL-Plus Zero-shot accuracy falls from 53.0% text-only to 50.6% multimodal.Text-only inputs show greater stability across many models and prompting strategies.
- Inference latency: Zero-shot prompting usually responds within 5 seconds, making it suitable for latency-sensitive applications.Most evaluated methods achieve real-time or near-real-time inference.
- Inference latency: CoT substantially increases latency, reaching approximately 22 seconds for Qwen3-VL-Plus and 13–14 seconds for Deepseek-V3.2 and Qwen3-Max.Self-Consistency requires 3–7 seconds, while memory-based methods remain below 2 seconds.
G. CoT Failure Cases
The failure cases show that CoT can overinterpret simple browsing behavior and generate unsupported assistance needs. This can produce false alarms even when zero-shot correctly predicts that no assistance is required.
- Model-dependent failures: CoT can lower performance on smaller open-source models, whereas larger models may gain from structured reasoning.Deepseek-V3.2 improves timing F1 from 69.5% to 71.3%, while Qwen3-VL-8B-Instruct experiences a dramatic accuracy drop.
- Model-dependent failures: CoT may improve semantic similarity for assistance content, such as Qwen3-Max rising from 0.285 to 0.305.The passage attributes this to better articulation of assistance content.
- False-positive case: In the multiple-tab case, zero-shot correctly predicts no assistance, but CoT invents information-overload concerns and predicts help incorrectly.The figure identifies this as a false-positive proactive-help prediction.
- False-positive case: Zero-shot or Self-Consistency is more robust when proactive systems must balance false alarms against coverage.This recommendation follows the observed volatility of complex reasoning chains.