Source-linked AI summary
SPILLage: Agentic Oversharing on the Web
Jaechul Roh, Eugene Bagdasarian, Hamed Haddadi, Ali Shahin Shamsabadi
TL;DR
Web agents act on live websites with access to personal resources, raising the question of whether they disclose task-irrelevant information through observable actions. The paper introduces SPILLAGE and benchmarks this risk across live e-commerce tasks, finding pervasive oversharing, with behavioral disclosure dominating content disclosure by 5×. Removing task-irrelevant information before execution improves task success by up to 17.9%.
Problem
The paper addresses whether web agents preserve contextual privacy while using personal resources to act for users on live websites.
Method
SPILLAGE provides a 2 × 2 taxonomy and benchmark for measuring explicit or implicit content and behavioral oversharing through step-level action analysis.
Results
Behavioral oversharing dominates content oversharing by 5× across 1,080 runs spanning two web-agent frameworks and three models.
Takeaways & Limitations
Protecting web-agent privacy requires evaluating observable actions alongside typed output, while removing task-irrelevant information can align privacy with utility.
Takeaways & Limitations
The evaluation focuses on OpenAI models, 180 e-commerce tasks, and single-website sessions.
Abstract
from arXiv · showhide
LLM-powered agents are beginning to automate user's tasks across the open web, often with access to user resources such as emails and calendars. Unlike standard LLMs answering questions in a controlled ChatBot setting, web agents act "in the wild", interacting with third parties and leaving behind an action trace. Therefore, we ask the question: how do web agents handle user resources when accomplishing tasks on their behalf across live websites? In this paper, we formalize Natural Agentic Oversharing -- the unintentional disclosure of task-irrelevant user information through an agent trace of actions on the web. We introduce SPILLage, a framework that characterizes oversharing along two dimensions: channel (content vs. behavior) and directness (explicit vs. implicit). This taxonomy reveals a critical blind spot: while prior work focuses on text leakage, web agents also overshare behaviorally through clicks, scrolls, and navigation patterns that can be monitored. We benchmark 180 tasks on live e-commerce sites with ground-truth annotations separating task-relevant from task-irrelevant attributes. Across 1,080 runs spanning two agentic frameworks and three backbone LLMs, we demonstrate that oversharing is pervasive with behavioral oversharing dominates content oversharing by 5x. This effect persists -- and can even worsen -- under prompt-level mitigation. However, removing task-irrelevant information before execution improves task success by up to 17.9%, demonstrating that reducing oversharing improves task success. Our findings underscore that protecting privacy in web agents is a fundamental challenge, requiring a broader view of "output" that accounts for what agents do on the web, not just what they type. Our datasets and code are available at https://github.com/jrohsc/SPILLage.
1. Introduction
The paper asks how web agents preserve user privacy while acting across live websites and introduces SPILLAGE to measure oversharing through both content and behavior. Benchmarking live e-commerce interactions, it finds oversharing is pervasive and that removing task-irrelevant information can improve task success.
- Motivation: Web agents may disclose task-irrelevant information through explicit or implicit content and behavioral actions during ordinary task execution.Behavioral traces include clicks, scrolling, and navigation patterns observable by websites.
- Framework: SPILLAGE characterizes agentic oversharing across directness—explicit versus implicit—and channel—content versus behavior.The framework captures both what an agent reveals and how it discloses that information to external parties.
- Benchmark: The benchmark evaluates natural oversharing on Amazon and eBay using mixed-context prompts and realistic input styles including chat history, email, and generic prompts.Tasks interleave task-relevant and task-irrelevant information and expose both content and behavioral interaction channels.
- Findings: +17.9% task success is achieved by removing task-irrelevant information before execution.This result indicates that higher web-agent utility does not require incurring oversharing.
- Findings: 1,080 agent runs across two frameworks and three GPT backbones show pervasive oversharing across tested configurations.The experiments include Browser-Use and AutoGen with o3, o4-mini, and gpt-4o.
2. Related Work
Web agents create privacy risks by acting on users’ behalf across live websites, while existing analyses primarily focus on content disclosures. This work targets non-adversarial oversharing arising from agents’ own task-execution behavior.
- Web agents actively interpret instructions, process web pages, and interact with live websites rather than operating only in closed, text-only environments.
- Privacy analysis of web agents concerns whether information flows to third-party services conform to contextual norms.
- Prior frameworks overlook behavioral oversharing and focus on explicit, verbatim content disclosures.
- The paper studies non-adversarial oversharing produced by an agent’s own task-execution behavior, without external attack or platform misconfiguration.
3. Problem Statement
The paper models web-agent privacy as a task-dependent information-flow problem: agents use personal resources to act on live websites, while passive observers infer unnecessary attributes from observable action traces.
- Users provide agents with personal resources and task instructions, creating an expectation that agents use only information required for the delegated task.
- An attribute is task-relevant when necessary for successful completion; other accessible attributes are task-irrelevant and should not be disclosed.
- SPILLAGE represents the user, agent, and passive observer as an information-flow setting involving resources, task instructions, and observable actions.
- The web action trace is the ordered sequence of observable operations the agent performs from task initiation through completion.
- A passive observer records the agent’s observable actions and infers task-irrelevant attributes without access to the original request.
4. SPILLAGE Framework
SPILLAGE characterizes web-agent oversharing across disclosure channel and directness, capturing both textual content and behavioral navigation. It defines an auditing framework and benchmark for measuring these categories on live e-commerce websites.
- Taxonomy: Explicit oversharing states task-irrelevant information verbatim, whereas implicit oversharing makes it inferable without stating it directly.The distinction applies to both textual content and behavioral navigation.
- Taxonomy: SPILLAGE categorizes oversharing by channel—content or behavior—and directness—explicit or implicit—yielding four categories.The categories are CE, CI, BE, and BI.
- Taxonomy: Content oversharing occurs through textual inputs, while behavioral oversharing occurs through clicks, filters, and scrolling.Behavioral traces can disclose information independently of text.
- Auditing implications: The two dimensions identify applicable defenses and distinguish information visible in search content from information revealed through logged navigation behavior.Text filtering and string matching address only subsets of the taxonomy.
- Auditing: The audit maps each action to oversharing events by identifying the task-irrelevant attribute and its content-or-behavior, explicit-or-implicit category.An LLM-based evaluator uses the original prompt, executed action, and declared next goal to produce structured annotations.
- Benchmark: The benchmark uses live Amazon and eBay tasks with ground-truth relevant and irrelevant attributes, realistic prompt styles, and freely chosen web navigation.Its design targets oversharing modes that existing task-success, text-leakage, and text-only benchmarks do not capture together.
5. Evaluation Results
Across 1,080 runs on live e-commerce sites, oversharing occurred in every tested configuration, with behavioral disclosures substantially exceeding content disclosures. Framework, prompt style, and model choice shaped the volume and form of risk, while removing task-irrelevant information improved task success.
- 1,080 runs across two frameworks and three models found oversharing in every tested configuration.
- Explicit oversharing: 905 behavioral versus 182 content oversharing events on Amazon with Browser-Use and gpt-4o show a 5× behavioral dominance.On eBay with AutoGen, the corresponding counts were 342 versus 46, a 7× difference.
- Framework effects: 0.684 [0.519, 0.860] behavioral oversharing rate on eBay for AutoGen exceeded Browser-Use’s 0.304 [0.224, 0.392].Browser-Use produced higher absolute occurrences through longer, fine-grained traces, whereas AutoGen concentrated risk per action.
- Prompt and model effects: 1.03 versus 0.37 behavioral oversharing rate on Amazon shows that generic prompts produced more oversharing than chat prompts with AutoGen.Generic prompts provided less conversational context for distinguishing task-relevant from task-irrelevant information.
- Implicit oversharing: 325 implicit content and 45 implicit behavioral oversharing events occurred on Amazon with Browser-Use and gpt-4o.Implicit oversharing was less frequent than explicit oversharing but remained substantial and could evade string-matching defenses.
- Defenses and utility: Up to +17.9% task-success gains followed removal of task-irrelevant prompt information, with Browser-Use accuracy rising from 73.4% to 99.4%.Both Browser-Use and AutoGen improved relative to their baselines after sanitization.
6. Discussion
The analysis attributes oversharing to both framework design and model-specific reasoning, while identifying important evaluation boundaries. The study uses OpenAI models, e-commerce tasks, and single-website sessions, although the taxonomy may extend to other domains.
- Frameworks do not separate task-relevant from task-irrelevant information before acting, and backbone models propagate context through different channels.The paper reports gpt-4o embedding preferences in queries, o3 surfacing details through actions, and o4-mini leaking through planning files.
- Limitations: The evaluation uses OpenAI models, so other model families may exhibit different oversharing patterns.A qualitative study of Brave AI Browsing, ChatGPT Atlas, and Perplexity Comet found wide variation in privacy preservation.
- Limitations: The 180-task evaluation targets e-commerce, though the taxonomy can be generalized to domains such as healthcare, legal services, travel, and finance.These domains also involve agents navigating external websites on users’ behalf.
- Limitations: Single-website sessions constrain the study because cross-site traces could enable richer inference attacks through behavioral patterns across third-party trackers.
7. Conclusion
SPILLAGE audits web-agent oversharing across content and behavioral channels and explicit and implicit forms. Across live e-commerce evaluations, behavioral oversharing dominated content by 5×, while sanitization improved task success by up to 17.9%.
- SPILLAGE uses a 2 × 2 taxonomy covering content and behavioral oversharing in explicit and implicit forms.
- 1,080 runs across two frameworks and three models on live e-commerce sites found pervasive oversharing, with behavioral oversharing dominating content by 5×.
- Up to 17.9% task-success improvement followed removal of task-irrelevant information before execution.The conclusion presents this result as evidence that privacy and utility are aligned.
- SPILLAGE extends privacy analysis beyond text to observable actions as a foundation for web agents that respect contextual integrity.
A. Analysis and Discussion
The paper distinguishes natural agentic oversharing from platform, network, and adversarial privacy threats. It argues that prior evaluations often miss behavioral channels and the explicit-versus-implicit distinction.
- SPILLAGE captures all four oversharing types on live websites, whereas prior work often covered only one channel or disclosure directness.
- The paper’s threat model focuses on information disclosed through the agent’s on-page actions, not platform configuration or network-layer metadata.
- Adversarial extraction studies assume attackers manipulate the agent’s environment, unlike the natural oversharing setting examined here.Examples include prompt injection, context hijacking, and reasoning-trace leakage.
- Text-only or binary detect-or-not evaluations miss behavioral leakage and the distinction between explicit and implicit disclosure.
A.2. Why Do Agents Overshare?
Agents overshare because frameworks process rich, context-heavy inputs without separating task-relevant from incidental personal information, while their action structures expose different opportunities for leakage. SPILLAGE evaluates these risks across content and behavioral channels, with explicit and implicit forms.
- Frameworks process rich user context without mechanisms to separate task-relevant from incidental personal information.
- Model reasoning architectures propagate user information differently: gpt-4o restates persona details, while o3 embeds preferences directly into actions.
- Browser-Use creates longer, fine-grained trajectories that increase behavioral exposure opportunities, whereas AutoGen compresses tasks into fewer, higher-level actions.Browser-Use actions tend to carry less information individually, while AutoGen concentrates more decision-making into each action.
- The formalization separates task-relevant attributes Sr from task-irrelevant attributes Si within user resources and represents execution as an observable action trace A.
- The evaluation judge inspects each agent action step for explicit and implicit oversharing across both content and behavioral channels.
B.5. Utility (Task Completion) Evaluation
Task-completion evaluation measures whether agents successfully complete assigned shopping tasks, using framework-specific completion procedures and persona-rich prompts containing task-irrelevant information.
- AutoGen task completion is evaluated with a prompt template, while Browser-Use logs success through built-in completion detection.
- The evaluation materials include generic-request, chat-style, and forwarded-email prompts with task-irrelevant information highlighted.
- Persona-rich examples include sensitive health details embedded in direct requests and chat histories.
C.1. Task Success Rates
AutoGen achieves higher and more stable task success than Browser-Use across the reported shopping evaluations, while shorter trajectories reduce failure and oversharing opportunities but can mask vulnerabilities during extended tasks.
- Browser-Use overall utility ranges from 0.742 to 0.761, while AutoGen exceeds 0.97 for o3 and o4-mini and reaches 0.861 for gpt-4o.
- AutoGen task-success scores range from 0.792–0.994, compared with 0.742–0.929 for Browser-Use.
- AutoGen’s streamlined orchestration requires fewer steps, reducing opportunities for task failure and producing more stable completion rates.
- Browser-Use produces more overall leaks through longer trajectories, while AutoGen shows higher per-step oversharing rates.On Amazon with o4-mini, Browser-Use produces 674 explicit behavioral and 382 explicit content leaks, while AutoGen reaches 0.852 explicit behavioral oversharing per step in the generic setting.
- Generic prompts consistently trigger higher implicit content and behavioral leaks, with Amazon showing more oversharing than eBay.
- Commercial-agent comparisons report privacy-preserving behavior for Brave AI Browsing and ChatGPT Atlas, contrasted with task-irrelevant information propagation by Perplexity Comet.
D. An Empirical Study of Oversharing in Commercial Web Agents
A manual study of three commercial web agents using persona-rich shopping prompts found sharply different privacy behaviors. Brave AI Browsing and ChatGPT Atlas avoided oversharing, whereas Perplexity Comet directly exposed sensitive user information to third-party sites.
- The study manually monitored three commercial agents across ten persona-rich shopping prompts without public APIs.
- Brave AI Browsing and ChatGPT Atlas consistently completed tasks using only task-relevant information and exhibited no oversharing.Their concise queries avoided propagating sensitive persona details.
- Perplexity Comet pasted large portions of user conversations into third-party search interfaces, disclosing trauma history, medication usage, and employer details.