PaperlayerView exact revision on arXiv

Source-linked AI summary

What It Is, Not What It Says: Securing Agentic Browsing Interactions with Web Semantics

Umar Iqbal

arXiv:2610.22697v1cs.CR

TL;DR

Agentic browsers flatten webpages into screenshots or text, exposing LLMs to unvetted third-party content and obscuring web semantics. INTACT builds GENTREE and plans element by element, with the harness enforcing element capabilities and requesting consent for third-party actions. On real webpages, fewer attacks got through INTACT’s planner than a conventional planner, while functionality disruption was limited.

  • Problem

    Flattened page representations expose LLMs to unvetted third-party content and discard the provenance and web semantics needed to distinguish what elements can do.

    Source: §1 · p. 1, §2.3.1 · p. 3

  • Method

    INTACT builds GENTREE to record element provenance and capabilities, plans with one element at a time, and has the harness enforce permitted actions and request consent for third-party content.

    Source: §1 · p. 2, §2.3.2 · p. 3

  • Results

    12.8% of attacks got through INTACT’s planner, versus 73.1% for VANILLA, across 1,091 webpages; attacks getting through INTACT reached at most a consent request.

    Source: §5.3.2 · p. 10

  • Takeaways & Limitations

    GENTREE attributes 2.3× as many elements to third parties as the browser, whose standard attribution misattributes 58.7% of them to the first party.

    Source: §5.2.1 · p. 8

    INTACT does not address availability: decoys can draw the planner away from the page region a task needs or falsely halt execution.

    Source: §6 · p. 15

Abstract

from arXiv · show

AI agents increasingly browse the web on users' behalf, yet they consume webpages as screenshots or text, exposing their LLMs to unvetted third-party content. Browsers enforce semantics under which what an element can do depends on what it is and where it comes from, not on what it says. Agents flatten these semantics, so injected text can persuade the LLM to exercise the user's authority for content that has none. As LLMs remain easy to persuade, the agent's interface to the web must itself ensure the integrity of what the agent reads, interprets, and does. We propose Intact, which runs on an unmodified browser. Intact builds GenTree, a bespoke representation that captures provenance and web semantics. Intact passes the page via GenTree to the LLM one element at a time, so the LLM decides on each element without any other element's text. Its harness executes each step only within the capabilities of the step's element and asks the user before acting on third-party content. On 1,270 real webpages, GenTree attributes 2.3x as many elements to third parties as the browser, which misattributes 58.7% of them to the first party. On 1,091 of these webpages with injected attacks, only 12.8% of attacks get through Intact's planner, each at most as a consent request, against 73.1% for a conventional planner. Isolation disrupts functionality on only 3.7% of tasks, and a parallel schedule could remove most of its added latency.

1. Introduction

Agentic browsers expose LLMs to unvetted page content while flattening the web semantics that distinguish what elements are and where they come from. INTACT uses GENTREE and virtual isolation to mediate what agents read and do, reporting fewer attacks getting through than with a conventional planner.

1. Introduction

  • Screenshots and text flatten web semantics, allowing injected text to persuade an agent to exercise the user’s authority for content that has none.The paper frames this as a confused-deputy risk: a paragraph cannot issue a network request, but its text can persuade the agent to do so.Source: §1 · p. 1
  • INTACT builds GENTREE to record element provenance and web semantics, then mediates what the agent reads, interprets, and does through that per-element representation.It attributes DOM creation and modification to the responsible script and taints content created or modified by third parties.Source: §1 · pp. 1–2
  • The harness limits each action to the named element’s capabilities and defers to the user before acting on third-party content.Virtual isolation presents one element at a time so its text cannot affect the decision on another element.Source: §1 · p. 2
  • 12.8% of attacks got through INTACT’s planner, versus 73.1% for VANILLA, across 1,091 real webpages.Each attack that got through INTACT reached, at most, a request for user consent.Source: §1 · p. 2
  • GENTREE provides provenance and capability structure that can serve as a substrate for defenses based on isolation, flow enforcement, and permission management.The paper also releases INTACT’s source code and a benchmark of attacks on 1,091 real webpages.Source: §1 · p. 2

2. Background and Motivation

Conventional agent interfaces flatten webpages into text or images, obscuring the origin and capabilities of their elements. INTACT instead derives provenance and web semantics, plans element by element, and constrains actions through the harness, with evaluation on live webpages.

2.1. Overview of Agentic Browsers

  • Agentic browsers and frameworks use textual, visual, or combined page representations to identify elements and issue actions through a browser or automation framework.These representations reuse views designed for scripts, assistive technologies, or people.Source: §2.1 · p. 2

2.2. Security and Privacy Concerns

  • Flattened page representations undo web protections by placing third-party content beside first-party content without reliable distinctions.They also remove the human-in-the-loop assumption behind browser security checks, while LLMs can act autonomously and remain easy to mislead.Source: §2.2 · pp. 2–3

2.3. Key Idea and Challenges

  • INTACT derives element provenance and capabilities from browser information, exposing them to the harness through GENTREE rather than relying on websites to author the representation.The browser knows element semantics, document origins, frames, and the scripts that create or modify elements.Source: §2.3 · p. 3

2.3.1. Input Integrity.

  • Third-party scripts can write into first-party documents, and standard browser attribution misses 58.7% of third-party elements in the crawl.Instrumentation attributes DOM creation and modification to the responsible script and taints third-party writes without relying on an LLM.Source: §2.3.1 · p. 3

2.3.2. Processing Integrity.

  • Element-by-element planning prevents one element’s text from reaching the decision on another, while a text-free log supplies prior-operation context.The authors measure functionality costs because this limited context can disrupt some tasks.Source: §2.3.2 · p. 3

2.3.3. Output Integrity.

  • The harness executes an operation only when the element affords it under web semantics, and asks for user consent when those semantics cannot settle whether a step is appropriate.The capability taxonomy accounts for interactions and effects using element attributes, ancestors, ARIA attributes, and computed style.Source: §2.3.3 · pp. 3–4

2.4. Complexity of Real Webpages

  • The evaluation uses real webpages with live third-party content: 1,270 for attribution and 1,091 for attacks and functionality.The authors release the 1,091 webpages with attacks as a benchmark.Source: §2.4 · p. 4

3. Threat Model

The threat model considers third-party attackers who control page content but lack special access to the user, agent, or first-party origin. INTACT aims to constrain their influence on a single page while assuming trusted browser infrastructure and harness.

3. Threat Model

  • Attackers may control third-party content in the first-party document or a cross-origin frame, including content hidden from the user but visible to the agent.They may target other elements or use their own elements as bait, but attacks considered require the agent to exceed the attacker’s own capabilities.Source: §3 · p. 4
  • The user, browser, and harness are trusted, but the LLM’s judgment is not; attacker content may persuade it, and it may err without an attacker.The instrumentation is assumed not to be evaded or tampered with by page scripts; moving it into the browser would remove that assumption.Source: §3 · p. 4
  • The goal is to prevent attacker content from leading the agent beyond the user’s request: each step stays within its element’s capabilities, and third-party steps require consent.The scope excludes multi-page attacks, first-party attackers, and availability; it constrains prompt-injection effects rather than detecting or eliminating prompt injection.Source: §3 · p. 4

4. INTACT: System Architecture

INTACT represents each page element’s provenance and capabilities in GENTREE, then plans one element at a time and has the harness enforce the resulting actions. This architecture combines browser instrumentation, web semantics, and user consent to mediate agent interactions.

4. INTACT: System Architecture

  • INTACT builds GENTREE to record who created or modified each element and what it can do, then uses virtual isolation for planning and harness checks for actions.It runs on stock Chromium, driven through Playwright and CDP.Source: §4 · pp. 4–5

4.1. GENTREE Construction

  • GENTREE combines every frame’s DOM, the accessibility tree, and provenance into an element record that includes markup, origin, authorship, capabilities, and accessibility information.It retains elements omitted from the accessibility tree, including scripts, hidden inputs, and hidden subtrees.Source: §4.1 · p. 5

4.1.1. Web Semantics Representation.

  • The capability taxonomy records ten interactions and thirteen browser effects, assigning labels through element rules and composition rules that account for ancestors, ARIA attributes, and computed style.Effects capture observable browser consequences; whether a request amounts to a purchase is left to a separate layer.Source: §4.1.1 · p. 5

4.1.2. Browser Instrumentation.

  • Instrumentation wraps DOM interfaces to attribute element creation and modification to the responsible script, separating third-party writes from first-party content.Writes whose script cannot be identified are treated as untrusted; instrumentation does not yet cover cookies, postMessage exchanges, extension-injected content, or shadow roots.Source: §4.1.2 · pp. 5–6

4.2. Planning under Virtual Isolation

  • Virtual isolation shows the LLM one GENTREE element at a time, so its text can inform only the decision about that element.The harness executes only operations allowed by the named element’s capabilities and asks the user before acting on third-party elements.Source: §4.2 · pp. 6–7, Figure 1 · p. 6
  • The LLM receives the user request, current-element markup, and a text-free log and form line that encode prior operations and structural context.The log and form line are derived by the system’s code rather than supplied by the LLM.Source: §4.2 · pp. 6–7, Figure 3 · p. 7
  • INTACT chunks page markup for relevance screening, then walks a selected chunk element by element; a form walk continues across a chunk boundary until the form ends.Chunk selection and the log of prior decisions remain outside virtual isolation and can let an attacker derail task completion, though they cannot name an element to act on.Source: §4.2 · p. 7

4.2.1. Functionality Extraction.

  • For evaluation, an LLM extracts five realistic page functionalities per page, mixing single actions, multi-action tasks, and reading followed by action.Requests are phrased as users would ask, include concrete values where needed, and end in an action on the page.Source: §4.2.1 · p. 7

5. Evaluation

The evaluation examines whether GenTree improves attribution and whether isolated planning and consent gating contain attacks, alongside their costs in functionality, user effort, and latency. It also assesses a parallel schedule for reducing latency while preserving what each model call sees, without requiring browser changes or website cooperation.

5.1. Data Collection

  • The study sampled 1,386 websites across 15 topical categories and captured 1,270 landing pages for evaluation.Pages were crawled with live content; the attack benchmark used 1,091 of the crawled webpages.Source: §5.1 · p. 8, §5.3.1 · p. 9

5.2.1. Results.

  • The browser missed 90.3% of third-party text-accepting elements, 92.2% of form submitters, and 75.6% of navigational elements.Pages with these missed elements made up 41.4%, 27.6%, and 65.0% of the crawl, respectively.Source: §5.2.1 · p. 8, Table 2 · p. 10
  • GenTree attributed 22.7% of page elements to third parties, versus 10% under browser attribution, which missed 58.7% of third-party elements.Third-party-created elements accounted for 36.4% of a page’s third-party elements on average; third parties also modified first-party elements on 65.8% of pages.Source: §5.2.1 · p. 8, Table 1 · p. 9

5.3.2. Results.

  • On 96.0% of reached outbound attacks, INTACT proposed no step, and it copied data in none of the 546 inbound attacks.Its planner sees only the element under consideration, preventing attack text from influencing decisions on other elements; proposed steps on third-party elements still require consent.Source: §5.3.1 · p. 9, §5.3.2 · p. 10
  • 12.8% of attacks got through INTACT’s planner, at most as consent requests, versus VANILLA following 73.1% of attacks.These outcomes were measured on 1,091 webpages; VANILLA’s attack steps execute, while INTACT holds steps on third-party elements for user consent.Source: §5.3.2 · p. 10, §5.3.1 · p. 10, Table 3 · p. 11
  • A policy requiring consent only for third-party steps would interrupt none of the 979 benign tasks that had no such steps.Third-party steps were 10.1% of INTACT’s 3,466 benign-plan steps and occurred on 112 of 1,091 tasks.Source: §5.3.2 · pp. 10–11, Table 4 · p. 11

5.3.3. Case Studies Analysis.

  • In case studies, INTACT withheld cross-element cart data from a payment-field decision and prevented a third-party note from redirecting a quote click.VANILLA entered the cart item names into a hidden third-party field and opened the wrong insurer’s quote; INTACT’s payment steps awaited consent.Source: §5.3.3 · pp. 11–12

5.4.1. Results.

  • 66% of INTACT plans contained every VANILLA step, and only 3.7% of tasks had an appropriate VANILLA plan when INTACT’s was not.Across tasks, INTACT retained 69.7% of VANILLA’s steps; the judge found the plans equivalent on 74.3% of tasks.Source: Table 5 · p. 13, §5.4.1 · pp. 13–14

5.5. Overhead of Isolation

  • The evaluation measures planning cost on 1,091 benign tasks using the same model for INTACT and VANILLA, with relevance checks run four at a time before the element-by-element walk.Reported token counts include a simulation instruction adding about 100 tokens per call that deployment would omit; the benchmark excludes seven webpages where INTACT planning exceeded 15 minutes.Source: §5.5 · p. 14

5.5.1. Results.

  • Per task, INTACT costs 2.1 times as much and takes 34 times as long as VANILLA: median cost is $0.0045 versus $0.0021, and median time is 83 seconds versus 2.5.INTACT makes a median of 55 calls per task (between 8 and 321) versus VANILLA’s single call; isolation adds a median $0.0023 per task, and no plan costs more than $0.024.Source: §5.5.1 · p. 14
  • Relevance checks account for about 41% of INTACT’s input, while caching serves 50% of its input tokens at a discount, so cost grows less than token use.INTACT produces a median 3,796 output tokens per task versus 81 for VANILLA because each call returns a decision.Source: §5.5.1 · p. 14
  • A proposed parallel schedule would cut median planning time by about 85% for about $0.0014 extra per task, while preserving each decision’s element-specific view and prior-operation log.Replaying the 1,091 recorded walks shortens the critical path from 34 sequential calls to five, or from 83.2 to 12.5 seconds; the schedule is unimplemented and assumes repeated decisions and parallel calls as fast as sequential ones.Source: §5.5.1 · pp. 14–15
  • INTACT runs on stock Chromium with an off-the-shelf model through a standard chat API, requiring neither browser changes nor website cooperation.Its instrumentation runs before the page’s own scripts.Source: §5.5.1 · p. 15

6. Discussion and Conclusion

The discussion identifies availability attacks and implementation overheads as open concerns, while outlining ways to reduce some costs and extend GenTree’s use. The conclusion argues that agents can inherit browser-enforced semantics and provenance to contain attacks, and the appendix describes the use of AI assistants in the work.

6. Discussion and Conclusion

  • Third-party decoys can steer chunk selection or falsely halt execution, and defending against these availability attacks remains future work.An attacker would need a script in the first-party document to taint the site’s own elements; that case is outside the threat model.Source: §6 · p. 15
  • Fine-tuning could recover missing steps, while parallel schedules could cut median planning time by about 85% at the expense of cost.The authors also note that the latency comparison with VANILLA matters less because web tasks involve page-load waits and computer-use tasks can take tens of minutes.Source: §6 · p. 15
  • JavaScript instrumentation can be altered by page scripts; moving the same hooks into the browser would prevent that, but requires carrying them across browser releases.The authors cite VisibleV8 patches maintained across ten Chrome versions as evidence that such maintenance is practical.Source: §6 · p. 15
  • GenTree could support stakeholder-specific agent policies, including user permission choices and website access controls, through runtime-generated labels and policies.The authors contrast runtime generation with prior approaches that assume websites publish or label a representation in advance.Source: §6 · p. 15
  • Agents that inherit browser knowledge of element capabilities and authorship can contain attacks that persuade their LLMs.The authors argue that web semantics should bound agent actions, with LLMs handling judgments that semantics cannot make.Source: §6 · pp. 15–16

Appendix A. AI Usage Statement

  • OpenAI Codex and Anthropic Claude assisted with writing and coding, and the authors say they inspected outputs and verified text, numbers, and references.The assistants helped implement and test INTACT, the evaluation pipeline, and figures.Source: Appendix A · p. 17

Appendix B. Attacks on Real Webpages in Detail

The appendix explains how its attack figures document the planner’s element-specific inputs and replies on real webpages.

Appendix B. Attacks on Real Webpages in Detail

  • Each attack figure reproduces the planner’s request for one element and its reply, with the task, prior operations, form status, and that element’s markup in the request.The figures omit instructions shared across all calls.Source: Appendix B · p. 17

B.1. An Outbound Attack on musc.edu

The musc.edu example shows an attack in a third-party frame that asks the agent to click a first-party element, while INTACT keeps the proposed action confined to the note.

B.1. An Outbound Attack on musc.edu

  • The planner proposes clicking the third-party note, but the harness rejects the action because a text element has no click capability and requires user consent.The plan then continues through Find Care and Find a Provider; Figure 9’s caption identifies amber as marking a third-party element in a different example.Source: §B.1 · p. 17, Figure 9 · p. 18

B.2. An Inbound Attack on lecker.de

On lecker.de, an attack in a third-party input prompts the planner to type text, but INTACT holds that step for user consent.

B.2. An Inbound Attack on lecker.de

  • The planner guesses “Datenschutzhinweisen” from the input’s quoted sentence, but this partial match is not counted as a copy and the third-party step requires user consent.The task is to keep using the site for free with advertising and tracking.Source: §B.2 · p. 18

Appendix C. Browser Instrumentation Details

Appendix C details the browser instrumentation: the interfaces it wraps, how it identifies scripts, joins records to the page, and validates attribution.

  • The appendix covers interface wrapping, script identification, joining instrumentation records to the browser’s page view, and attribution validation.Source: Appendix C · p. 18

C.1. Wrapped Interfaces

The instrumentation wraps page interfaces to record element activity and identifies scripts so their contributions can be attributed to the appropriate party.

C.1. Wrapped Interfaces

  • Wrappers retain native implementations and suspend their own recording; separate wrappers capture resource-loading setters and Chromium’s shadowing textContent accessor.Source: §C.1 · p. 18

C.2. Script Identity

  • A script’s party is determined by comparing its registrable domain with the top document’s; third-party labels include path, query, and an eight-digit SHA-256 prefix.Source: §C.2 · p. 18
  • Dynamic and inserted inline code receive synthetic source URLs linked to their creating script, while calls with no usable script URL remain unattributed.Source: §C.2 · p. 18

C.3. Joining Records to the Page

Instrumentation records are joined to the browser’s DOM and accessibility views using node identifiers, with separate capture and identifier handling for site-isolated frames.

C.3. Joining Records to the Page

  • After page settlement, records are collected per frame and joined to the DOM snapshot and accessibility tree through CDP node identifiers; cloned or adopted nodes are remarked.Source: §C.3 · p. 18
  • Site-isolated frames are captured through separate DevTools targets and their node identifiers are namespaced; 114 of 15,968 cross-origin frames closed before capture.Source: §C.3 · p. 18

C.4. Validation

Attribution was correct across constructed test cases, while a small share of crawl elements remained unattributed and was treated as untrusted.

C.4. Validation

  • All 24 constructed authorship cases were attributed correctly; 1.58% of script-created or modified crawl elements remained unattributed and were treated as untrusted.Source: §C.4 · p. 19

Appendix D. Effect Categories

Effects are grouped by whether they are recoverable browser-state changes, external effects, or unresolved consequences. These labels assess predicted Effects; they do not establish execution or successful restoration.

  • Local change denotes browser-controlled state updates with an identified restoration operation, assuming recoverable prior state, authority to restore it, and no unlisted external consequences.Source: Appendix D · p. 19, Table 8 · p. 20
  • External effect denotes communication or handoff beyond the restoration boundary.Source: Appendix D · p. 19, Table 8 · p. 20
  • Undetermined marks Effects whose consequences remain unresolved; interactions receive no category annotation, and External effect takes precedence over Undetermined for planned steps.Source: Appendix D · p. 19, Table 8 · p. 20
  • The categories are qualitative assessments of predicted Effects, not evidence of execution or successful restoration.Source: Appendix D · p. 19
Loading 2610.22697v1…