Source-linked AI summary
The Verifiable Action Card: Trustworthy Human-in-the-Loop Control for Secure Autonomous Agents
Hasnain Irshad, Anam Mughees, Neelam Mughees, Abdullah Mughees, Imtiaz Ali Soomro
TL;DR
Agentic browsers can act with users’ authority while processing attacker-controlled page content, and approval prompts may themselves be influenced by untrusted text. VAC reconstructs approval from the ground-truth action and intent provenance, presents it in trusted browser chrome, and binds it to the action re-verified at dispatch. In evaluation, aggregate attack success fell from 78% to 0% on every evaluated model.
Problem
Agentic browsers can be redirected by attacker-controlled page content, while approval descriptions may rely on model-generated or page-controlled text rather than the pending action.
Source: §1 · p. 1Method
VAC derives approval from the ground-truth pending action and trusted intent provenance, renders it out-of-band, and rechecks the action at dispatch.
Source: §4.5 · p. 7, §14 · p. 16Results
78% aggregate attack success without VAC fell to 0% on every evaluated model.
Source: §14 · p. 16Takeaways & Limitations
A human-in-the-loop control is only as trustworthy as the evidence it shows the human.
Source: §14 · p. 16Evaluation uses a transparent oracle rather than recruited users and one DOM-first agent with a bounded scenario suite, so absolute numbers are not claimed to transfer to every agent, model, or site.
Source: §11 · pp. 15–16
Abstract
from arXiv · showhide
Agentic browsers can execute security-sensitive actions under a user's authenticated session, making indirect prompt injection and deceptive confirmation interfaces a direct threat to action integrity. Existing human-in-the-loop (HITL) safeguards are insufficient when the approval prompt itself can be influenced by untrusted page content or model-generated text. We present the \emph{Verifiable Action Card} (VAC), an architectural defence that reconstructs approval information from the ground-truth pending browser action and trusted intent provenance, renders it out-of-band in the trusted browser chrome, and binds approval to the exact action re-verified at dispatch. VAC combines provenance fencing, a ground-truth action descriptor, default-deny confirmation, provenance-aware risk gating, and execution binding. We implement VAC in a complete agentic browser and evaluate it on a 24-scenario benchmark covering confused-deputy attacks, Lies-in-the-Loop dialog forging, indirect prompt injection, adaptive action substitution, provenance evasion, and legitimate tasks. Across the evaluated LLMs, attack success without VAC ranges from $68\%$ to $100\%$, whereas VAC reduces attack success to $0\%$ on every model, with $78\%$ legitimate-task completion and a $0\%$ false-block rate. These results show that grounding approval in the action that will actually execute provides architectural protection against security failures that prompt-level defences and conventional HITL confirmation cannot reliably prevent.
1. Introduction
Agentic browsers expose users to attacks that exploit page content and deceptive approval prompts, because the agent acts with the user’s authority while conventional confirmations can misrepresent the pending action. VAC grounds approval in the true action and intent provenance, and its cross-model evaluation reports zero attack success on every evaluated model.
1. Introduction
- Agentic browsers can be redirected by attacker-controlled page content, and conventional HITL prompts can be forged so users approve actions different from those they believe they authorize.The agent may consume visible or hidden page text while acting with the user’s authority.Source: §1 · p. 1
- VAC reconstructs approval from the ground-truth pending action and intent provenance, presents it in trusted browser chrome, and binds approval to the action re-verified at dispatch.Its five components include provenance fencing, a ground-truth descriptor, default-deny confirmation, provenance-aware risk gating, and execution binding.Source: §1 · p. 2
- 80–100% attack success without VAC fell to 0% on every evaluated model, supporting the claim that its protection is architectural rather than tied to a specific LLM.The evaluation ran identically across five open-weight models.Source: §1 · p. 2
2. Background and Related Work
- The browser-agent literature emphasizes capability, while VAC addresses the distinct question of preventing adversarial pages from turning agent capability against the user.A DOM-first action space makes its ground-truth descriptor practical to compute.Source: §2 · p. 3
- VAC instantiates consent integrity for browsers by reading the action from the DOM, rendering approval out-of-band, and re-verifying the action at dispatch, unlike concurrent work with an assumed trusted path.The paper reports implementing and evaluating the mechanism with live LLMs across multiple models.Source: §2 · p. 4
- VAC complements reasoning-level defenses: those reduce how often malicious actions are attempted, while faithful confirmation checks any attempted action against ground truth.The evaluation harness scores server-recorded side effects and varies the confirmation arm to measure security-utility costs.Source: §2 · p. 4
3. System and Threat Model
The threat model treats the browser agent’s authenticated authority and irreversible actions as assets, while allowing an attacker to control page content and assuming the LLM may be compromised. It frames the central failure as approval based on attacker-influenced descriptions rather than the true action and trusted intent.
3.1. System model and assets
- The system protects the user’s authenticated authority and the integrity of irreversible actions in a single-user browser agent.The modeled system includes the page, perception module, LLM, executor, and browser chrome.Source: §3.1 · p. 4
3.2. Attacker capabilities and goals
- Attackers may control visible or attribute-based page content and may induce visits, but cannot modify the user instruction, agent code, executor, browser chrome, or host OS.The model treats the LLM as potentially compromised and does not trust its confirmation text.Source: §3.2 · pp. 4–5
3.3. Trust boundaries
- For confirmation integrity, the LLM is on the untrusted side of the boundary; deriving approval evidence from it would place the attacker inside the decision’s trusted computing base.Source: §3.3 · p. 5
3.4. Attack classes
- The attack classes include malicious forms that silently redirect submissions and dialog forging that disguises a commit with a benign page-controlled label.Source: §3.4 · p. 5
3.5. Why current HITL fails
- Current HITL can fail when a malicious action’s true parameters are paired with a benign description derived from page labels or model text.The proposed defense computes the description from true action parameters and trusted intent, then renders it on a channel the attacker cannot alter.Source: §3.5 · p. 5
4. Design: The Verifiable Action Card
VAC makes approval depend on a trusted description of the pending action and its intent provenance, then verifies at dispatch that the action still matches what was approved. Its components combine provenance fencing, a ground-truth descriptor, a default-deny card, risk gating, and execution binding.
4. Design: The Verifiable Action Card
- VAC’s invariant is that no irreversible action executes without approval of a ground-truth, provenance-aware description shown on a channel the page cannot alter.The design combines five independently testable components.Source: §4 · p. 5
4.1. C1: Intent provenance
- C1 labels user-origin and page-origin content so C4 can mechanically enforce provenance, rather than relying on prompt fencing to persuade the model.Prompt-level fencing alone can be evaded.Source: §4.1 · p. 6
4.2. C2: Ground-truth action descriptor
- C2 reads the resolved element and enclosing form to record the verb, destination, recipient, amount, file path, and accessibility name.The descriptor reflects the element the executor is about to act on, not its displayed label or a model-written summary.Source: §4.2 · p. 6
4.3. C3: Out-of-band, default-deny confirmation
- C3 renders the structured descriptor in trusted browser chrome, displays parameter provenance, warns on elevated risk, and makes Deny the prominent default.The page cannot access the chrome to restyle, pad, hide, or forge the card.Source: §4.3 · p. 6, Figure 2 · p. 7
4.4. C4: Provenance-aware risk gating
- C4 detects commits structurally and marks them elevated risk when a recipient or amount is unattributed, or when a read-only task attempts a commit.Fully user-attributed commits receive a lighter single-tap confirmation.Source: §4.4 · p. 7
4.5. C5: Execution binding
- C5 rechecks approved material parameters at dispatch and aborts with a renewed card if the outgoing recipient or amount differs.The comparison normalizes whitespace and case for recipients and digits for amounts, so only material changes trigger it.Source: §4.5 · p. 7
4.6. How the components compose
- The components address distinct attacks: C1 discourages injections, C2 and C4 expose hidden recipients, C2 and C3 defeat forged dialogs, and C5 catches post-approval substitutions.The design also pairs C2’s ground-truth recipient with C4’s domain-level attribution check against look-alike recipients; no single component is load-bearing alone.Source: §4.6 · p. 8
4.7. Security analysis
- VAC’s residual avenues include social engineering that induces the user to name an attacker’s address and heuristic attribution errors that can weaken detection or prompt needless confirmation.Systematic evaluation of adversarial page-mutation scenarios is left to future work, and residual risk is bounded by the executor’s view of the action.Source: §4.7 · pp. 8–9
5. Worked Examples
The worked examples show how VAC changes the evidence presented at sensitive actions: it exposes hidden destinations and misleading labels, and flags commits prompted by injected page instructions.
5.1. Confused-deputy exfiltration form
- A hidden external recipient turns a benign contact-form submission into exfiltration; VAC reveals the recipient and raises elevated risk, while a naive “Submit” confirmation is approved.The user denies VAC’s card, which defaults to Deny.Source: §5.1 · p. 9
5.2. Dialog forging (Lies-in-the-Loop)
- A “Save draft” button conceals a transfer to mule@evil.com for 5000; the naive label-based summary is approved, while VAC exposes the true action and the user denies it.The descriptor marks the recipient and amount page-origin and the action elevated risk.Source: §5.2 · pp. 9–10
5.3. Indirect instruction injection
- For an informational task, injected page text may prompt an extra commit or override; VAC fences it as page-origin and flags any attempted commit as unrequested.The naive arm catches a blatant Delete instruction but misses a plausible “Verify” request.Source: §5.3 · p. 10
6. Implementation
VAC is implemented as localized additions to an open agentic browser, with components for provenance, descriptor construction, risk gating, and trusted confirmation. The implementation also logs commit decisions and outcomes for audit.
6. Implementation
- The implementation pairs an Electron/Chromium shell with a Python agent using Playwright over CDP, indexed DOM elements, and configurable OpenAI-compatible models.Its additions are described as modest and localized.Source: §6 · p. 10
6.1. Concrete artifacts
- C1 fences page-derived observations, C2 reads action fields from the resolved element, C4 gates commits, and C3 displays an escaped card with Deny as the default focus.The card is rendered in the renderer process; each commit’s descriptor, decision, and outcome are recorded for audit.Source: §6 · p. 10, §6.1 · p. 10
- The gate is deterministic and requires no additional model call; C2 performs one DOM evaluation per commit, while C1 and C4 use local string operations.The dominant per-step cost remains the LLM inference common to all arms.Source: §6.1 · p. 10
7. Experimental Methodology
The evaluation compares four confirmation arms across attack and legitimate scenarios, using a per-arm oracle and server-side outcomes. It runs the same benchmark protocol across models, with both full-suite repeated runs and a smaller pilot.
7.1. Arms
- The four arms are no-gate, naive page-label confirmation, VAC, and VAC without provenance; A-versus-C tests the architectural claim, while B and C-p isolate confirmation and provenance contributions.All four arms are run across the models and seeds of the sweep.Source: §7.1 · p. 10
7.2. Scenario suite
- The 24-scenario suite contains 13 attacks across four classes and 11 legitimate commit tasks, including hidden recipients, inflated amounts, forged labels, substitutions, look-alike addresses, and injected text.The scenarios use self-contained local pages; Table 7 is identified as a representative subset of the full suite.Source: §7.2 · pp. 10–11, Table 7 · p. 11
7.3. Confirmer oracle
- A transparent per-arm oracle models user responses: the naive confirmer denies alarming visible actions but can be fooled by forged-benign labels, while the VAC confirmer denies elevated-risk actions.The VAC oracle sees the ground-truth descriptor and risk.Source: §7.3 · p. 11
7.4. Metrics
- ASR and TCR measure security and utility, while Conf measures confirmation burden and FB measures legitimate commits wrongly denied; outcomes use server-side ground truth.Attempt rate distinguishes provenance’s effect on attempts from the gate’s effect on reached commits.Source: §7.4 · p. 11
7.5. Setup and reproducibility
- The primary protocol runs four arms across five open-weight models; two models receive the full 24-scenario suite with three seeds, and three others appear in a single-seed subset pilot.The reported sweep contains 293 clean trials, plus a separate 40-trial GPT-4.1 run for breadth; conclusive outcomes are required for trial inclusion.Source: §7.5 · p. 11
8. Results
VAC reduces attack success across evaluated models, including attacks that resemble legitimate tasks or adapt to the confirmation process. The discussion compares VAC with naive confirmation, describes its adaptive-attack protections, and reports legitimate-task outcomes.
8. Results
- 78% legitimate-task completion with a 0% false-block rate accompanied VAC’s 0% attack success across the five models.VAC averaged 0.94 confirmations per legitimate task.Source: §8 · p. 12
- Disguised confused-deputy and Lies-in-the-Loop attacks succeeded without a gate and were blocked by VAC; overt injections were partly resisted by model safety.The disguised attacks included contact exfiltration, inflated payment, and a transfer; delete/verify injections were partly resisted.Source: §8 · p. 12, Figure 5 · p. 14
8.1. Naive HITL, ablation, and adaptive attacks
- Naive confirmation left attack success at 72%, versus 78% with no gate; VAC reduced it to 0% at 78% task completion.The naive and VAC arms used the same human and provenance; the reported difference was what the human saw. On Lies-in-the-Loop transfer, naive HITL was fooled in 100% of trials while VAC blocked all.Source: §8.1 · p. 12
- Disabling provenance fencing still held attack success at 0%, indicating that the structural commit gate and ground-truth descriptor carried the security in this ablation.The authors characterize provenance fencing as a defence-in-depth signal rather than the load-bearing element.Source: §8.1 · p. 12
- VAC blocked both post-approval recipient substitution and look-alike-recipient provenance evasion, which succeeded without a gate.Execution binding caught the changed outgoing recipient; provenance-aware attribution caught an address whose address or domain was absent from the instruction.Source: §8.1 · pp. 12–13, Table 9 · p. 13
8. Results
- 68%–100% baseline attack success fell to 0% with VAC across the five evaluated open models.The reported comparison uses the same benchmark protocol across models; GPT-4.1 is a separate preliminary run.Source: §8 · p. 11, Table 8 · p. 12, Figure 4 · p. 14
9. Discussion
The discussion frames VAC as a ground-truth confirmation layer that complements reasoning-level defences and can apply where an executor resolves actions before execution. It also identifies residual risks and bounds the evaluation’s claims.
Discussion
- VAC moves the trust anchor from agent narration to the action itself, applying separation of designation from authority at the decision-to-act boundary.Even a compromised reasoning step cannot alter the resolved target, recipient, or amount the executor reads, or restyle the card in a process the page cannot reach.Source: §9 · p. 15
- VAC complements reasoning-level defences: those reduce malicious-action attempts upstream, while VAC checks any attempted commit against ground truth.The authors describe this as defence-in-depth rather than a replacement for spotlighting or capability systems.Source: §9 · p. 15
- The mechanism assumes an executor that can resolve the concrete action and a trusted UI surface; the descriptor schema could extend to filesystem or API actions.The proposed fields include path, endpoint, and method.Source: §9 · p. 15
- VAC does not prevent read-only misinformation, navigation side-channel leaks, or malicious actions a user is socially engineered into instructing.The authors characterize VAC as a last line for irreversible effects, not a universal shield.Source: §9 · p. 15
- The authors argue that confirmation evidence should come from the resolved action and appear on a trusted surface, not from model- or page-supplied text.They propose this as a checkable design rule for agent-security guidance and platform review.Source: §9 · p. 15
- The evaluation uses a transparent human oracle, one DOM-first agent, and a bounded attack set; it does not measure real users’ catch rates or establish generality across agents and models.The authors say a controlled user study should establish catch rates and limit their claim to consistency across evaluated models.Source: §9 · p. 15
10. Deployment Considerations
VAC adds no model calls: its descriptor uses a DOM evaluation and its gate is deterministic. Deployment discussion addresses human confirmation burden, configurable policy, and applicability to agents whose executors can resolve actions.
Deployment Considerations
- VAC adds no model calls; its descriptor is a single DOM evaluation and its gate is deterministic, leaving per-step latency and token cost indistinguishable from the undefended agent.Added latency is the human decision time, incurred only on commits; routine user-attributed commits require a single tap.Source: §10 · p. 15
- VAC scales prompts with risk by confirming structurally detected commits and reserving the prominent banner card for elevated-risk actions.Fully user-attributed commits receive a lightweight confirmation; the confirmation-burden metric makes this cost explicit for policy tuning.Source: §10 · p. 15
- The risk policy is a small, auditable rule set operators can inspect or tighten, and audit logs support later review and incident response.Examples include treating any cross-origin POST as elevated or adding domain allow-lists.Source: §10 · p. 15
- The descriptor abstraction applies wherever an executor can resolve an action before performing it, including DOM-first browsers and, with a richer schema, computer-use and tool-calling agents.The richer schema would include endpoint, method, and path.Source: §10 · p. 15
11. Threats to Validity
The validity discussion separates interface isolation from generalization: the oracle makes the B-versus-C comparison fair under a diligent-user assumption, while the evaluation scope limits transfer of absolute rates.
Internal validity
- The per-arm oracle assumes a diligent user; real catch-rates require human-subject study, although identical oracle behavior across B and C fairly isolates confirmation faithfulness.A distracted or habituated user could weaken every arm.Source: §11 · p. 15
Construct validity
- Server-side submissions measure actual effects rather than agent self-reports, while confirmation-burden and false-block metrics capture usability costs.The server-side scoring correctly measures claims of success without action, or trials blocked after acting.Source: §11 · pp. 15–16
External validity
- The evaluation covers one DOM-first agent and a bounded ten-scenario suite, so absolute results are not claimed to transfer to every agent, model, or site.The authors identify a larger scenario corpus and wider model set as future work.Source: §11 · p. 16
Reliability
- An early run was invalidated by shared-port cross-contamination; the revised pipeline uses ephemeral ports, excludes errored trials, deduplicates records, and supports re-scoring from raw logs.The released artifact regenerates every number from raw logs.Source: §11 · p. 16
12. Ethics and Responsible Disclosure
The conclusion reports aggregate attack success falling from 78% to 0% on every model while preserving utility, with VAC grounding out-of-band confirmation in action and intent provenance. The surrounding sections bound the evaluation to local pages and identify human validation and broader generalization as future work.
12. Ethics and Responsible Disclosure
- All experiments used local, self-hosted pages; no third-party site was attacked, and no real funds, accounts, or personal data were involved.The released benchmark reproduces publicly documented techniques without releasing a novel attack capability.Source: §12 · p. 16
13. Future Work
- Future work includes human-subject measurement of catch-rates and fatigue, stronger parameter taint tracking, broader agent and tool tests, and adaptive-attacker red-teaming.The authors also plan a hardened TOCTOU-resistant dispatch path.Source: §13 · p. 16
14. Conclusion
- VAC reconstructs confirmation from the ground-truth action and intent provenance, then renders it out-of-band with default-deny behavior.The conclusion describes this as restoring the guarantee that confirmation reflects the action that will execute.Source: §14 · p. 16
- The method reduced aggregate attack success from 78% to 0% on every model while preserving utility; its post-decision check makes protection architectural rather than tied to a particular LLM.Source: §14 · p. 16
A. Scenario Specifications
- Attack pages hide malicious fields behind a benign visible control, while structurally similar legitimate pages omit those fields and use authorized destinations or values.Each scenario is a self-contained HTML page whose local server records submissions as ground truth.Source: A. Scenario Specifications · p. 16
B. Example Agent Trace
The appendix shows the VAC arm pausing at commit, specifies how the artifact regenerates and audits trials, and defines per-arm approval and server-side outcome scoring.
B. Example Agent Trace
- The VAC trace completes the visible contact-form task but halts at commit, where the gate detects a post-form submission.The trace is identified as a confused-deputy contact-form example.Source: B. Example Agent Trace · p. 16
C. Reproduction
- The reproduction artifact runs scenario-arm-seed trials, regenerates tables and figures, logs descriptors and outcomes, and enables independent re-scoring.Ephemeral ports, exclusion of errored trials, and deduplication reduce interference and duplicate records.Source: C. Reproduction · pp. 16–17
D. Confirmer Oracle and Scoring
- The naive oracle checks visible summaries for alarming unrequested verbs, while VAC denies elevated-risk actions; server-recorded submissions determine attack and legitimate-task outcomes.Neither the oracle nor scorer consults the agent’s narration.Source: D. Confirmer Oracle and Scoring · p. 17