Source-linked AI summary
ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions
Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, Zhenyu Yan
TL;DR
Existing proactive agents have limited open-world perception and functionality, motivating a context-aware approach to understanding user intentions and deciding when assistance is needed. ContextAgent combines wearable sensory and persona contexts with LLM reasoning and tool calling, and ContextAgentBench evaluates the resulting task across daily scenarios. It outperforms six baselines on proactive prediction and tool-calling measures, while the benchmark and tool set remain bounded in scope.
Problem
Existing proactive agents have limited open-world perception and restricted functionality for proactive services, leaving context-aware proactive assistance insufficiently evaluated.
Method
ContextAgent extracts sensory and persona contexts, uses a context-aware reasoner to predict proactive-service needs, and sequentially calls external tools when assistance is warranted.
Results
ContextAgent outperforms six baselines by up to 8.5% in proactive-prediction accuracy, 7.0% in tool-calling F1-score, and 6.0% in tool-argument accuracy.
Takeaways & Limitations
Wearable sensory context and tool-augmented reasoning support the development of more ubiquitous proactive AI assistants.
Takeaways & Limitations
The current tool set contains twenty API-based tools, and ContextAgentBench covers nine life scenarios that could be further diversified.
Abstract
from arXiv · showhide
Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent.
1 Introduction
Existing LLM agents are largely reactive or limited to enclosed environments, while ContextAgent uses open-world wearable perceptions, persona context, and LLM reasoning to provide tool-augmented proactive assistance. ContextAgentBench evaluates this task across daily scenarios and tools, with ContextAgent outperforming six baselines.
- Most LLM agents initiate tasks only after explicit user instructions and lack autonomous environmental perception for proactive support.
- Existing proactive agents reduce reliance on instructions but remain limited in open-world perception and functionality for proactive services.
- Wearable devices such as smart glasses and earphones provide hands-free open-world perceptions that can support automatic intention understanding while reducing user workload.
- ContextAgent extracts sensory and persona contexts, reasons about the need for assistance, and calls external tools when proactive service is warranted.
- 1,000 samples across nine daily scenarios and twenty tools form ContextAgentBench, enabling comprehensive evaluation of context-aware proactive agents.
- Up to 8.5% higher proactive-prediction accuracy, 7.0% higher tool-calling F1-score, and 6.0% higher tool-argument accuracy are reported against six baselines.
2 Related Works
Prior work spans reactive LLM agents and emerging proactive agents, but existing approaches generally rely on user queries, enclosed computer interfaces, or direct inference rather than rich open-world sensory context.
- Reactive LLM-agent research targets tasks including web navigation, software engineering, personal assistance, and robotics, alongside planning and tool-use capabilities.
- Proactive-agent research removes or reduces explicit-query requirements across coding, conversation, clarification, and multi-agent cooperation settings.
- Existing proactive approaches commonly observe computer interfaces or re-ask users for information, without using rich sensory contexts to initiate services.
- Existing benchmarks mainly evaluate reactive agents, while ProactiveBench is limited to desktop UI environments and omits multimodal sensor context.
3 Context-aware Proactive Agent Task
The context-aware proactive-agent task maps multimodal wearable perceptions and persona information to reasoning traces, proactive decisions, tool chains, and responses. ContextAgentBench operationalizes this task with annotated contexts, scores, planned tools, and broad daily-life coverage.
- 3.1 Task Definition: The task takes multimodal wearable perceptions and persona context as input and produces thought traces, a proactive score, tool chains, and a final response.
- 3.1 Task Definition: A user-adjustable threshold triggers assistance only when the proactive score reaches PS ≥ θ; otherwise, the agent does not disturb the user.
- 3.2 Task Construction: ContextAgentBench is designed as the first benchmark for evaluating context-aware proactive LLM agents.
- 3.2 Task Construction: Each dataset sample includes sensory context, persona, thought traces, proactive score, planned tool chains, and response information.
- 3.2 Task Construction: Sensory context covers visual information, acoustic information, and smartphone notifications collected from egocentric wearable perspectives.
- 3.2 Task Construction: Proactive scores range from 1 to 5, with scores of 1 or 2 indicating that no tool chain or response is needed.
- 3.2 Task Construction: The benchmark contains 1,000 samples spanning nine daily-life scenarios and twenty tool types, with each sample potentially involving up to five tools.
4 ContextAgent Framework
ContextAgent extracts proactive-oriented sensory and persona contexts, then reasons over them to predict whether assistance is needed and execute tool chains when appropriate.
- ContextAgent extracts proactive-oriented contexts from multimodal sensory perceptions before integrating them for tool-augmented proactive services.
- Sensory context captures surroundings and actions, while persona context represents personal information such as past behaviors, preferences, and identity.
- The system uses in-context learning with vision-language models to derive proactive-oriented visual contexts from egocentric videos rather than relying on zero-shot descriptions.
- Persona contexts complement sensory observations by helping the agent determine the need and urgency of proactive assistance and can be updated from historical sensory data.
- The context-aware reasoner maps sensory context and personas to thought traces, proactive scores, and tool chains, initiating services when PS ≥θ.
- ContextAgent fine-tunes the reasoner with distilled thought traces so it generates explicit reasoning before proactive predictions and tool calls.
5 Experiments
Experiments evaluate ContextAgent across proactive prediction, tool calling, modality and persona ablations, reasoning traces, model settings, tool-chain lengths, and out-of-distribution scenarios. ContextAgent generally leads baselines, while removing sensory modalities, personas, or reasoning traces reduces performance.
- Evaluation setup: The evaluation measures proactive prediction with Acc-P, missed detections, false detections, and RMSE, alongside tool-calling precision, recall, F1-score, and Acc-Args.
- Evaluation setup: The benchmark comparison includes Proactive Agent, multiple ICL variants, CoT, and SFT baselines under a 60% training and 40% testing split.
- Benchmark results: 8.5% higher Acc-P, 7.0% higher F1-score, and 6.0% higher Acc-Args are achieved by ContextAgent over baselines on ContextAgentBench with Llama3.1-8B-Instruct.
- Benchmark results: 6.2% Acc-P, 3.0% F1-score, and 7.6% Acc-Args improvements over the best baseline are achieved on ContextAgentBench-Lite with Qwen2.5-7B-Instruct.
- Ablation study: Missing vision or audio decreases Acc-P by up to 17.9% and F1-score by up to 23.3%, with missing vision having the larger impact.
- Ablation study: Removing personas decreases Llama-3.1-8B-Ins performance by up to 9.0% in Acc-P, 12.3% in F1-score, and 12.6% in Acc-Args.
- Ablation study: Thought traces improve ICL-All by up to 20.1% in Acc-P compared with ICL-P, although their benefits are reduced under SFT.
- Generalization: Under out-of-distribution evaluation, ContextAgent reaches up to 90.9% Acc-P, 68.9% F1-score, and 51.6% Acc-Args, outperforming the best baseline by 8.3%, 10.7%, and 1.9%, respectively.
6 Conclusion
The paper presents ContextAgent and ContextAgentBench as steps toward proactive AI assistants that use rich wearable sensory context to support tool-augmented reasoning.
- ContextAgent harnesses extensive sensory perceptions and tool-augmented LLM reasoning to provide enhanced proactive services.
- ContextAgentBench evaluates context-aware proactive LLM agents and supports the paper’s move toward proactive AI assistants using hands-free wearable sensors.
A Data Examples
The examples show ContextAgent combining sensory observations with persona information to decide whether proactive assistance is warranted and which tools to call. Its prompts instruct the agent to reason before acting and avoid assistance when user intent or necessity is insufficiently clear.
- Examples: ContextAgent uses visual, audio, and persona information to infer user intent and determine whether proactive assistance is appropriate.In the hiking example, conversation about organizing a team-building hike leads to reasoning about weather, location, timing, and transportation.
- Examples: A hiking scenario maps contextual reasoning to a tool chain including GPS, weather, datetime, and agenda-conflict checks.The planned response uses the resulting information to offer practical assistance for the weekend activity.
- Examples: For a user with diabetes, persona information and meal context motivate health-data retrieval and a knowledge search before dietary guidance is provided.The resulting response recommends more salad, less pasta, and recognizes grilled chicken as having minimal impact on blood sugar.
- Prompts: The system prompt supplies sensory and persona contexts, requires a thought and proactive-service decision, and restricts tool selection to specified tools with correct arguments.The prompt explicitly says to propose assistance only when the user’s actions are fully understood.
C Dataset Details
The dataset is built by scaling human-designed exemplars through an automated diversification pipeline and pairing textual contexts with raw sensory data. Data collection includes privacy safeguards and informed consent.
- Dataset construction: Annotators create examples across nine everyday scenarios, document reasoning and tool chains, and cross-review samples to reduce overproactivity and annotation errors.This process produces 200 human-created exemplars for the seed dataset.
- Dataset construction: An automated diversification pipeline expands the seed data into ContextAgentBench and ContextAgentBench-Lite, while raw video and audio are collected to match textual sensory descriptions.The raw data comes from copyright-free internet sources and the authors’ own collections.
- Data governance: The study reports IRB approval, informed consent from participants, and blurring of sensitive video regions such as faces.These measures are described as privacy protections for collected participant data.
D Tool Definition
ContextAgentBench defines twenty external tools for connecting sensory context to tool-based agent behavior. Its context-extraction prompt directs models to produce objective first-person visual descriptions from wearable-camera video.
- Tool definition: ContextAgentBench contains 20 tools with documented names, descriptions, and input and output arguments.The tools are adopted from existing work to support the benchmark’s focus on sensory-context-to-tool-agent connections.
- Context extraction: The proactive-oriented extraction prompt asks a vision-language system to analyze video frames and generate detailed, objective Vision-field descriptions from the user’s first-person perspective.The descriptions represent what smart glasses or another egocentric camera would capture.
E More Results
Across benchmark, lite, out-of-domain, and ablation analyses, ContextAgent consistently performs strongly against baselines, including when using smaller models and when multiple tools are required. Results also identify benefits from thought traces, persona context, and sensory context.
- Benchmark results: ContextAgent’s strongest baseline comparison reaches 92.1% Acc-P, 71.5% F1-score, and 59.6% Acc-Args for ICL-All using GPT-4o.The cited results also report comparable performance from ContextAgent with a 7B-parameter model.
- Benchmark results: ContextAgent consistently outperforms baselines using the same base LLMs and can approach the performance of systems using 70B-parameter models.With Qwen2.5-7B-Ins, it is 0.7% lower in Acc-P, 1.9% higher in F1-score, and 16.9% higher in Acc-Args than the best Llama3.1-70B-Ins baseline in the cited comparison.
- Ablation and analysis: Thought traces improve few-shot prompting, with ICL-All outperforming ICL-P most of the time and larger models benefiting more than 7B models.Using Llama3.1-70B-Ins under OOD settings, ICL-All is 20.1% higher in Acc-P, 20.7% higher in F1-score, and 14.3% higher in Acc-Args than ICL-P.
- Ablation and analysis: Models from 1.5B to 3B parameters perform only 2.9%, 3.9%, and 5.5% lower than 7B models on Acc-P, F1-score, and Acc-Args, respectively.The result suggests potential deployment on mobile devices without cloud access.
- Ablation and analysis: Removing persona context reduces Acc-P by up to 12.0% and Acc-Args by up to 14.3%, while sensory context perception and thought traces also provide positive benefits.The ablation identifies persona context as especially important across the reported metrics.
- Tool-chain analysis: For level 3 samples requiring 3–5 tools, ContextAgent with Qwen2.5-7B-Ins achieves 30.3% higher Acc-P and 16.0% higher Acc-Args than baselines.The analysis reports particularly strong performance in scenarios requiring multiple external tools.
F Limitations
The paper identifies limitations in ContextAgent’s current tool coverage and benchmark scope, while pointing to standardized tools and broader scenarios as future directions.
- The current ContextAgent tool set contains twenty tools with APIs.
- ContextAgentBench currently covers nine daily-life scenarios, leaving room for further diversification.
- The authors suggest integrating MCP-standardized tools to improve tool-call standardization and tool-type diversity.
G Broader Impacts
ContextAgent uses wearable sensory data and tool-based reasoning, with evaluation materials including ablations and results across three sample levels.
- ContextAgent uses sensory data from wearable devices such as smart glasses and earphones.
- The ablation study distinguishes removing persona information from removing thought processes during supervised fine-tuning.
- ContextAgentBench reports separate results for Level-1, Level-2, and Level-3 samples.