PaperlayerView exact revision on arXiv

Source-linked AI summary

ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions

Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, Zhenyu Yan

arXiv:2505.14668v2cs.AIcs.CLcs.HC

TL;DR

Existing proactive agents have limited open-world perception and functionality for proactive service. ContextAgent uses wearable sensory contexts and user personas to reason about when proactive services are needed and call tools; on ContextAgentBench, it outperformed six baselines across proactive prediction and tool-calling metrics.

  • Problem

    Existing proactive agents have limited ability to perceive open-world environments and restricted functionality for proactive service.

    Source: §1 · pp. 1–2

  • Method

    ContextAgent reasons over sensory contexts from wearable perceptions and user personas to predict the need for services and generate tool chains for proactive assistance.

    Source: §3.1 · p. 4, §4.2 · p. 6

  • Results

    Against six baselines, ContextAgent achieved up to 8.5% higher proactive-prediction accuracy, 7.0% higher tool-calling F1-score, and 6.0% higher tool-argument accuracy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The work takes a step toward proactive AI assistants by leveraging rich context from hands-free wearable sensors to enhance proactive LLM reasoning.

    Source: §6 · p. 10

    The evaluation scope is bounded by a 20-tool API set and nine life scenarios that the authors say could be expanded.

    Source: F Limitations · pp. 24–25

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent.

1 Introduction

ContextAgent addresses proactive agents’ limited open-world perception and functionality by extracting sensory and persona contexts, predicting when help is needed, and calling tools; ContextAgentBench evaluates this capability.

1 Introduction

  • Prior programming agents [24] [55] depend on inputs such as screenshots or keyboard actions, whereas ContextAgent targets open-world daily-life perception through hands-free wearables.Reactive agents also require explicit user instructions, while prior proactive systems have limited open-world perception and functionality.Source: §1 · pp. 1–2
  • ContextAgent extracts sensory and persona contexts from egocentric video and audio, then reasons about service necessity and calls external tools when needed.Its reasoner uses reasoning traces distilled from advanced reasoning LLMs to support thinking before acting.Source: §1 · pp. 2–3
  • ContextAgentBench contains 1,000 samples across nine daily-life scenarios and twenty tools; the evaluation compares six baselines and tests 13 LLMs.The scenarios include working and chitchat.Source: §1 · p. 3
  • Up to 8.5% higher accuracy for proactive predictions, 7.0% higher F1-score for tool calling, and 6.0% higher accuracy for tool arguments versus six baselines.The evaluation compares ContextAgent with six baselines on ContextAgentBench.Source: §1 · p. 3

2 Related Works

Prior LLM agents are largely reactive or rely on constrained forms of proactivity, while existing benchmarks do not evaluate proactive assistance grounded in wearable sensory context and diverse tool use.

Reactive LLM-based Agents

  • Prior LLM agents span task domains and capability improvements, but generally require explicit user instructions and do not use wearable sensor perceptions for proactive assistance.Work includes web navigation [12] [56], software engineering, and personal assistants [44] [42], alongside planning, function calling, reflection, generalization, and collaboration research.Source: §2 · p. 3

Proactive LLM Agents

  • Existing proactive agents use re-asking, teammate-action prediction, or desktop-interface observations, rather than rich wearable sensory context, and primarily infer directly instead of integrating external tools.Ask-before-plan [52] still requires an initial query; ProAgent [50] focuses on teammates, while Proactive Agent [24] and CodingGenie [55] monitor computer interfaces.Source: §2 · p. 3

LLM Agent Benchmark

  • Existing benchmarks mainly evaluate reactive instruction-following, while ProactiveBench [24] is limited to desktop UI and direct inference rather than wearable context and diverse tool calling.The related-work discussion identifies a gap in comprehensive evaluation of proactive reasoning grounded in wearable sensory information with tool-calling capabilities.Source: §2 · p. 3

3 Context-aware Proactive Agent Task

Section 3 defines a proactive LLM agent that combines wearable sensory perceptions with persona context to decide whether assistance is needed and, if so, call tools. It introduces ContextAgentBench and describes how its context-rich samples and annotations support evaluation.

3.1 Task Definition

  • The agent combines wearable sensory perceptions and user personas to produce reasoning traces, a proactive score, planned tool chains, and a final response.It initiates assistance only when the score meets a user-adjustable threshold; otherwise, it does not disturb the user.Source: §3.1 · p. 4

3.2 Task Construction

  • ContextAgentBench is introduced as the first benchmark designed to evaluate context-aware proactive LLM agents.Its design emphasizes wearable sensory context, diverse personas, and tool-using assistance rather than direct inference alone.Source: §3.2 · p. 4
  • Each sample contains seven components: sensory perceptions, context, persona, thought traces, proactive score, planned tool chains, and final response.Annotators describe visual, acoustic, and smartphone-notification context and include persona information when relevant.Source: §3.2 · p. 4
  • Annotators assign proactive scores from 1–5 and specify tool chains for cases needing assistance; scores of 1 or 2 have no tool chain or response.The available tool set includes predefined names, descriptions, arguments, and formats.Source: §3.2 · p. 4
  • The dataset contains 1,000 samples across nine daily-life scenarios and 20 tool types, with up to five tools per sample.ContextAgentBench-Lite adds 300 human-verified samples paired with raw sensor data, including egocentric video and, for some conversations, audio.Source: §3.2 · p. 5

4 ContextAgent Framework

ContextAgent turns wearable sensory perceptions into proactive-oriented contexts, combines them with user personas, and uses an LLM reasoner to decide whether to provide tool-augmented assistance.

4.1 Proactive-oriented Context Extraction

  • ContextAgent combines visual, audio, and smartphone-notification contexts with persona information to support user-intent inference and proactive predictions.It uses in-context learning with vision-language models to generate visual contexts, speech recognition for audio, and can update persona contexts from historical sensory data such as daily conversations [44].Source: §4.1 · pp. 5–6

4.2 Context-aware Proactive Reasoning

  • The context reasoner takes sensory and persona contexts to generate thought traces, proactive scores, and tool chains, initiating sequential tool execution when the score reaches threshold θ.It is trained with supervised fine-tuning on chain-of-thought traces distilled from advanced LLMs, enabling think-before-act reasoning; tool results are integrated with the contexts and traces to generate final responses.Source: §4.2 · p. 6

5 Experiments

ContextAgent leads the reported benchmark comparisons, while ablations show the importance of sensory and persona contexts and thought traces; it also retains strong performance under scenario-based OOD evaluation.

5.1 Experimental Setup

  • Experiments compare ContextAgent with Proactive Agent, ICL and SFT variants across 13 proprietary, open-source, and small LLMs, using a 60%/40% train-test split.ICL baselines use 10 demonstrations; variants add thought traces and/or personas, while SFT-P uses sensory and persona contexts.Source: §5.1 · p. 7
  • Evaluation covers proactive prediction with Acc-P, MD, FD, and RMSE, and tool calling with Precision, Recall, F1-score, and Acc-Args.Acc-Args checks tool names and arguments, counts a sample incorrect if any argument is wrong, and is calculated only for correctly predicted tools.Source: §5.1 · p. 7

5.2 Results on Benchmarks

  • 8.5% in Acc-P, 7.0% in F1-score, and 6.0% in Acc-Args are ContextAgent’s improvements over baselines with Llama3.1-8B-Instruct on ContextAgentBench.ContextAgent achieves the highest performance across all reported metrics; against baselines using 70B-scale and proprietary LLMs, its Acc-P is -1.5%, F1-score is -3.0%, and Acc-Args is +6.6%.Source: Figure 5 · p. 7, §5.2 · p. 7, Table 1 · p. 8
  • ContextAgent combines sensory and persona contexts to select proactive services, map contexts to tool chains, and personalize predictions or decide not to interrupt.Examples include checking GPS, datetime, bus schedules, and ride-booking options near a bus station; assessing outdoor plans with weather and agenda tools; and offering health-related restaurant suggestions for health-conscious users.Source: §5.2 · p. 8, Figure 7 · p. 9, Figure 8 · p. 9

5.3 Ablation Study and Discussion

  • Missing vision or audio lowers Acc-P by up to 17.9% and F1-score by up to 23.3%, while removing personas lowers Llama-3.1-8B-Ins Acc-P, F1-score, and Acc-Args by up to 9.0%, 12.3%, and 12.6%.Missing vision has a larger impact than missing audio. Zero-shot Qwen-2.5-VL context extraction lowers Acc-P, F1-score, and Acc-Args by 3.0%, 3.3%, and 1.9%, respectively; the paper attributes the gap to missing proactive-oriented cues.Source: §5.3 · pp. 8–9, Table 3 · p. 10

5.2 Results on Benchmarks

  • 6.2% Acc-P, 3.0% F1-score, and 7.6% Acc-Args are ContextAgent’s improvements over the best baseline on ContextAgentBench-Lite with Qwen2.5-7B-Instruct.Both ContextAgent and baselines show slight performance degradation on Lite, but ContextAgent remains highest across all metrics and is comparable to or better than 70B-scale and proprietary-model baselines.Source: §5.2 · p. 7, Figure 6 · p. 8, Table 2 · p. 9

5.3 Ablation Study and Discussion

  • 20.1% improvement in Acc-P is achieved by ICL-All over ICL-P, while thought-trace benefits are reduced under SFT.Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct perform comparably and outperform DeepSeek-R1-7B; across tool-chain lengths, most approaches have higher MD but lower FD under prompts encouraging conservative service initiation.Source: §5.3 · p. 9

5.4 Out-of-Domain Evaluation

  • 90.9% Acc-P, 68.9% F1-score, and 51.6% Acc-Args are achieved by ContextAgent under scenario-based OOD evaluation, outperforming the best baseline by 8.3%, 10.7%, and 1.9%, respectively.Training uses six scenarios and evaluation the remaining three; ContextAgent is also reported as comparable to proprietary and advanced reasoning LLMs, with stronger LLM reasoning further improving prediction and tool-calling accuracy.Source: Figure 9 · p. 10, §5.4 · p. 10, Table 4 · p. 10

6 Conclusion · Appendix

The conclusion presents ContextAgent as a framework that combines extensive sensory context with tool-augmented LLM reasoning for proactive services, and introduces ContextAgentBench to evaluate this task.

6 Conclusion

  • ContextAgent is introduced as the first framework for context-aware proactive LLM agents.The paper frames it as a step toward proactive AI assistants that use rich context from hands-free wearable sensors.Source: §6 · p. 10
  • ContextAgent harnesses extensive sensory context and tool-augmented LLM reasoning to enhance proactive services.Source: §6 · p. 10
  • ContextAgentBench is introduced as the first benchmark for evaluating context-aware proactive LLM agents.Source: §6 · p. 10

A Data Examples · B Prompts

The examples illustrate how sensory context and user personas guide proactive assistance or restraint, while the prompt descriptions explain how ContextAgent, its baselines, and the data-generation pipeline are specified.

A Data Examples

Example 1

  • A road conversation about a team hike prompts location and condition checks using GPS, weather, date-time, and agenda-conflict tools before an offer of help.The response reports clear skies and mild temperatures and asks whether the user wants packing help or suggestions.Source: Example 1 · p. 14

Example 2

  • A dining scene, diabetes history, and slightly elevated glucose motivate tailored dietary advice, including more salad or less pasta and an offer of further help.The example frames the advice as useful without being intrusive.Source: Example 2 · pp. 14–15

Example 3

  • A weather example links short sleeves and shorts to a forecasted temperature drop after 6 PM, prompting a recommendation to add layers.The agent checks the user’s location and local forecast before offering the reminder.Source: Example 3 · pp. 15–16

Example 4

  • With limited persona information about a gym user, the example assigns a proactive score of 2 and provides neither tools nor a response.The reasoning says more fitness-level information could enable more accurate proactive services.Source: Example 4 · p. 16

Example 5

  • For a user walking uphill, the example finds no explicit need for action and assigns a proactive score of 1 with no tools.The context also mentions trees along the road and a dog-owner persona, but no further information prompting assistance.Source: Example 5 · p. 16

B Prompts

  • ContextAgent’s prompt combines fixed task instructions and tool definitions with runtime persona and context; baseline prompts vary few-shot examples and persona inclusion.Proactive Agent has no prompt examples; Vanilla ICL, CoT, ICL-P, and ICL-All use ten training samples with progressively different context or thought content, while Vanilla SFT omits persona context.Source: B Prompts · p. 17
  • Data generation uses scenario-aware and proactive-score-aware strategies, specifies sample structure and tools, and draws on multiple LLMs to increase sample diversity.Generation prompts include persona and completed-sample examples and guide scenario- and score-specific patterns.Source: B Prompts · pp. 17–18

C Dataset Details

ContextAgentBench is built from annotator-designed seed exemplars and an automated diversification pipeline, alongside raw video and audio data tied to sensory-context descriptions. The data comes from copyright-free internet platforms and the researchers’ own collections, with participant consent and video privacy protections.

C Dataset Details

  • Annotators designed seed exemplars, which an automated diversification pipeline scaled to generate ContextAgentBench and ContextAgentBench-Lite.Figure 14 is described as showing the pipeline flowchart.Source: C Dataset Details · p. 18
  • Raw video and audio data were collected based on the dataset’s textual sensory-context descriptions from copyright-free internet platforms and the researchers’ own collections.Self-collected samples were primarily used for chitchat scenarios.Source: C Dataset Details · pp. 18–19
  • The study received IRB approval, participants gave informed consent before data collection, and sensitive video regions such as faces were blurred.Source: C Dataset Details · p. 19

D Tool Definition

ContextAgentBench defines 20 tools for bridging sensory context with tool-based LLM agents.

D Tool Definition

E More Results

ContextAgent outperforms same-base-model baselines across benchmark settings, with 7B performance comparable to 70B baselines. Further analyses show benefits from thought traces, persona and sensory contexts, and performance on longer tool chains, alongside examples of appropriate proactive and non-proactive behavior.

Full Results on Benchmark

  • 92.1% Acc-P, 71.5% F1-score, and 59.6% Acc-Args are achieved by ICL-All with GPT-4o, the highest-performing baseline on ContextAgentBench.ContextAgent consistently outperforms baselines using the same LLMs; with a 7B LLM, it is 0.4% lower on Acc-P, 1.5% lower on F1-score, and 0.4% higher on Acc-Args than the best 70B baseline.Source: E More Results · p. 21, Table 6 · p. 22

Impact of Thought Traces on ICL

  • 20.1% higher Acc-P, 20.7% higher F1-score, and 14.3% higher Acc-Args are achieved by ICL-All than ICL-P with Llama3.1-70B-Ins on ContextAgentBench OOD settings.CoT generally outperforms Vanilla ICL, and ICL-All usually outperforms ICL-P; thought traces offer greater benefits for 70B than 7B LLMs, which may have limited capacity to learn distilled traces.Source: E More Results · p. 22

Impact of Different Base Models

  • 2.9%, 3.9%, and 5.5% lower Acc-P, F1-score, and Acc-Args, respectively, are observed for 1.5B–3B LLMs compared with 7B LLMs.The results suggest potential mobile deployment without cloud access, which could further reduce privacy concerns and system overhead [41].Source: E More Results · p. 22, Table 8 · p. 23

Ablation Study

  • 12.0% lower Acc-P and 14.3% lower Acc-Args are observed at most when persona context is removed from ContextAgent.Persona context is crucial; sensory context perception and thought traces also provide positive benefits.Source: E More Results · p. 23

Performance Across Different Tool Chain Lengths

  • 30.3% higher Acc-P and 16.0% higher Acc-Args are achieved on level 3 samples with Qwen2.5-7B-Ins than by baselines.Across tool-chain lengths of 0–1, 2, and 3–5 tools, ContextAgent achieves significantly higher F1-score and Acc-Args than baselines on level 2 and level 3 samples.Source: E More Results · pp. 23–24

Full Results on ContextAgentBench-Lite and OOD Settings

  • 0.7% lower Acc-P, 1.9% higher F1-score, and 16.9% higher Acc-Args are achieved by ContextAgent with Qwen2.5-7B-Ins versus the best Llama3.1-70B-Ins baseline on ContextAgentBench under OOD settings.ContextAgent consistently achieves the highest performance against baselines using the same base LLMs and remains comparable to 70B baselines on ContextAgentBench-Lite and OOD settings.Source: E More Results · pp. 21–22, Table 9 · p. 24

More Qualitative Results

  • ContextAgent initiates support when it detects contexts such as product interest or loading a washing machine, and withholds support during leisure or ordinary activities.External information such as online product prices and app-based recommendations can improve service usefulness; withholding unnecessary support helps reduce interruptions.Source: E More Results · p. 24

F Limitations

ContextAgent’s limitations concern the scope of its tools and benchmark scenarios, with MCP integration and broader scenario coverage identified as future directions.

F Limitations

  • ContextAgent currently includes twenty API tools, and integrating MCP could standardize tool calls and diversify tool types.MCP is described as aiming to standardize external tools.Source: F Limitations · pp. 24–25
  • ContextAgentBench covers nine life scenarios, and further diversification could improve ContextAgent’s capabilities and practicality for daily use.Source: F Limitations · p. 25

G Broader Impacts

The section frames ContextAgent as a first context-aware proactive LLM-agent framework that combines wearable sensory context with tool-based reasoning, and presents its benchmark and research links as steps toward human-centric proactive assistants.

  • ContextAgent combines rich sensory context with tool-based LLM reasoning to improve perception and functionality for proactive service.The paper presents this as its first exploration of context-aware proactive LLM agents.Source: G Broader Impacts · p. 25
  • Hands-free, egocentric sensing through wearables can improve understanding of users’ conditions and intentions while reducing cognitive and physical workload.Examples of devices include smart glasses and earphones.Source: G Broader Impacts · p. 25
  • ContextAgentBench is introduced as the first benchmark for context-aware proactive LLM agents, while ContextAgent connects ubiquitous mobile sensory-context research with emerging LLM agents.The paper presents these contributions as opening new research perspectives and supporting human-centric proactive AI assistants.Source: G Broader Impacts · p. 25

Related papers

Loading 2505.14668v2…