PaperlayerView exact revision on arXiv

Source-linked AI summary

Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?

Ting Yan

arXiv:2608.27443v1cs.HCcs.CR

TL;DR

The paper asks whether reusable, user-authored consequence policies improve control of AI agents acting across digital services for non-expert users. It compares three permission designs in a controlled study and finds that POLICY reduced overreach protection despite lowering runtime prompts, because users mostly chose ask and approved many overreach actions at runtime.

  • Problem

    Non-expert users need understandable, reusable ways to control agents across services, but it is unclear whether standing policies improve protection compared with per-action decisions.

  • Method

    The study compared per-action human approval, automated model review, and user-authored allow, ask, or never rules over consequence categories with 113 non-expert participants.

  • Results

    POLICY blocked less overreach than HITL and AUTO, while reducing runtime prompts relative to HITL; across all 7 overreach actions, POLICY had the highest approval rate.

  • Takeaways & Limitations

    User-authored rules did not necessarily create stronger limits because participants mostly chose ask, returning decisions to runtime where many overreach actions received approval.

  • Takeaways & Limitations

    The simulated one-day study used online participants without professional software backgrounds and no real-world consequences, limiting generalizability to long-term use, experienced users, and real decisions involving personal data or money.

Abstract

from arXiv · show

AI agents are poised to become a primary interface to digital products, acting across email, files, payments, and personal data. People without professional software backgrounds need understandable, reusable ways to control actions across services. We examine a mechanism in which a language model maps actions to plain-language consequence categories with user-authored "allow", "ask", or "never" rules. We ask what is gained and lost when decisions are made in advance as reusable rules rather than separately for each action. We analyzed 113 participants without professional software backgrounds across three conditions: per-action human-in-the-loop approval (HITL), automated per-action model review (AUTO), or user-authored consequence policy (POLICY). Participants judged 2 examples in each of 4 consequence categories; POLICY participants then set one rule per category. All supervised an 18-action simulated day, including 7 overreach actions. POLICY blocked less overreach than HITL (-20.1 percentage points, 95% CI [-32.1, -8.1]) and AUTO (-14.5 points, 95% CI [-25.8, -3.2]). POLICY lowered runtime prompts from 18.0 to 10.9, but total intervention time was not reliably lower when rule setup was included. Exploratory analysis showed that participants chose "ask" for 114 of 140 POLICY rules, returning most overreach actions to runtime. Of the 148 overreach actions executed in POLICY, 133 followed human approval and 15 ran automatically under "allow" rules. Across all 7 overreach actions, POLICY had the highest approval rate. Counterintuitively, user-authored rules did not by themselves provide stronger protection: many actions outside users' original requests went through after users approved them. These results reveal a gap between preference and commitment: repeatedly choosing "ask" preserves case-by-case choice but prevents a standing policy from settling decisions in advance.

1 Introduction

AI agents increasingly act across heterogeneous digital services, creating a need for understandable permission controls for non-experts. This study compares per-action approval, automated review, and user-authored consequence policies to examine protection, burden, and rule use.

  • AI agents are becoming general-purpose interfaces for ordering, messaging, purchasing, and managing files across many tools.
  • Consequence-level policies let users apply allow, ask, or never rules across tools based on recognizable consequences rather than technical operations.
  • Participants selected ask for most rules, so many POLICY decisions returned to runtime instead of becoming settled standing permissions.
  • 113 participants without professional software backgrounds were compared across HITL, AUTO, and POLICY conditions in a controlled study.
  • POLICY blocked less overreach than both baselines and did not reliably reduce total intervention time, although it reduced runtime permission prompts relative to HITL.
  • The paper contributes an MCP proxy and a human-labeled 120-tool evaluation set for connecting consequence rules to tool execution.

2 Background & Threat Model

Modern agents discover and invoke tools from multiple services through MCP, while permission decisions rely on metadata that may be unreliable or malicious. This creates risks from both changing tool sets and opaque classification.

  • MCP lets host applications discover tools through tools/list and invoke them through tools/call over JSON-RPC.
  • Agents can discover heterogeneous tools at runtime, making technical operations difficult for users to evaluate consistently.
  • The proxy’s classifier reads tool metadata that may be misleading, malicious, or insufficient to reveal actual capabilities.

3 Related Work

Prior work identifies a tension between repeated human approval, opaque automated review, and reusable permissions that may not fit later contexts. This paper studies that tension for non-experts using consequence-level rules.

  • Prior systems commonly force a choice between repeated approval and opaque automated review, while advance permissions may fail to fit later contexts.
  • Existing approaches express permissions over tools, arguments, resources, provider hierarchies, or MCP servers rather than user-recognizable consequences.
  • Commercial coding agents already combine reusable rules, approval modes, automated review, and sandboxing; this study instead targets non-professional users.
  • Prior permission-reuse research shows that a decision suitable for one case may not fit a later case.
  • Usable-security research finds that permission decisions are context-dependent, while repeated notices impose cumulative burden.
  • The study adapts impact and reversibility concepts into consequence categories and treats additional agent-initiated actions as overreach.

4 User-Facing Consequence Categories

The study groups heterogeneous tools by consequences users can recognize, then lets participants govern four categories with allow, ask, or never rules. The taxonomy separates action type, data sensitivity, and externality.

  • Consequence categories allow one rule to govern actions from different tools that share what they do and affect.
  • The taxonomy separates action type, data sensitivity, and externality, preserving distinctions such as reading a bank record versus reading nonsensitive data.
  • The user study used four plain-language categories: spend money, send or publish information, delete something, and access private information.
  • Five actions fell outside the four categories and ran automatically, allowing the study to distinguish settled rules from runtime asks.
  • The corpus contained 538 traceable tools from 36 MCP servers, with a 120-tool evaluation set labeled independently by three human annotators.
  • Human annotator agreement was high, but labels assessed metadata-classifiable capabilities rather than verified runtime behavior.

5 Mapping Tools and Applying Rules

The proxy maps discovered tools to consequence labels, then applies users’ allow, ask, or never rules when tools are invoked. The study used fixed researcher-checked mappings, separating rule behavior from live classification errors.

  • The proxy classifies each discovered tool, caches its consequence labels, and applies matching rules when the tool is invoked.It forwards, prompts, or blocks the call based on the resulting decision.
  • Rules map consequence labels to allow, ask, or never, with the most restrictive matching decision taking priority.Missing rules, invalid outputs, failures, and low-confidence classifications trigger a runtime prompt.
  • The user study used four simpler categories and fixed, researcher-checked routes for every scripted action rather than live classification.Mapper variation therefore could not create differences among study conditions.

6 Evaluating the Mapper

The mapper was evaluated against human labels on unseen and broader tool sets, with additional tests of simpler methods, repeated-run consistency, and rule effects from classification errors.

  • 6.1 Accuracy on unseen tools: 69.0% exact-set agreement and 90.0% main-action agreement were achieved on 100 unseen tools.Agreement was 77.8% for data sensitivity and 86.9% for externality on this intentionally challenging set.
  • 6.1 Accuracy on unseen tools: The prompted LLM had higher action-label agreement than three simpler mapping methods on all 120 tools.This full-set comparison includes the 20 tools used to refine its instructions, so it is descriptive rather than held-out.
  • 6.2 Mapper errors and their effects: Five difficult tools changed action labels across repeated runs, while clear tools received the same labels every time.Among difficult tools, four also varied on data sensitivity and five on externality across ten classifications each.
  • 6.2 Mapper errors and their effects: A simulated never rule failed to stop 14% of tools in the blocked category and stopped 4% outside it.Routing low-confidence results to ask reduced pass-through to 11% but increased unnecessary prompts to 7%.
  • 6.2 Mapper errors and their effects: The evaluation used ordinary tool metadata and did not test intentionally misleading descriptions or prompt injection.The mapper therefore was not evaluated against malicious metadata conditions.

7 User Study Method

The study compared per-action human approval, offline model review, and user-authored standing rules during a common 18-action simulated day. It separated task authorization from personal preference and measured protection, completion, prompts, and intervention time.

  • 7.1 Study design and conditions: The three conditions differed in how permission decisions were made while all participants supervised the same simulated day and saw the same action descriptions.The designs were HITL, AUTO, and POLICY.
  • 7.1 Study design and conditions: AUTO decisions were generated offline before the study, with 8 actions allowed and 10 escalated, including all 7 beyond-task actions.No live model classification occurred during participant sessions.
  • 7.2 Simulated day: The 18-action day contained 11 required actions and 7 overreach actions across morning, afternoon, and evening scenes.Actions mapped to zero or more categories: spending, sending or publishing, deletion, and private-data access.
  • 7.3 Participants and procedure: The sample comprised 113 participants without professional software backgrounds recruited from U.S. English-speaking residents.The study used online participation and a scripted simulation rather than real-world consequences.
  • 7.4 Measures and analysis: The study evaluated overreach blocking and required-action completion separately, while intervention burden included runtime prompts and POLICY rule-setup time.Analyses accounted for repeated decisions by participants and differences among actions.
  • 7.1 Study design and conditions: POLICY participants authored four consequence-category rules before the simulated day, whereas HITL and AUTO used per-action decision timing.Pre-task preferences were not used to configure conditions or define correctness.

8 Results

Across the controlled study, POLICY blocked less overreach than HITL and AUTO while preserving high required-action completion. It reduced runtime prompts, but its rules usually returned decisions to runtime, where participants often approved overreach.

  • Study design: 113 participants without professional software backgrounds were analyzed across HITL, AUTO, and POLICY conditions.The study compared per-action human approval, automated model review, and user-authored consequence policies.
  • Required-action completion: 94.1%–96.9% was the required-action completion range across conditions, with neither adjusted comparison showing a clear difference.The study did not test whether completion rates were equivalent.
  • Overreach blocking: 20.1 percentage points lower than HITL and 14.5 points lower than AUTO was POLICY’s adjusted overreach-blocking rate.The confidence intervals were [-32.1, -8.1] versus HITL and [-25.8, -3.2] versus AUTO; only the HITL difference remained reliable after correction.
  • Runtime approvals: POLICY had the highest runtime approval rate for all 7 overreach actions.Its approval rate was 26.4 percentage points higher than HITL’s and 20.6 points higher than AUTO’s, conditional on actions routed through ask.
  • Rule choices: 81.4% of POLICY rules were ask rules: participants chose ask for 114 of 140 rules, leaving only 26 decisions settled in advance.Ask was the most common choice in every consequence category.
  • Decision paths: 133 of 148 executed POLICY overreach actions followed affirmative human approval, while 15 ran automatically under allow rules.Ask routed 199 actions to runtime prompts; participants approved 133 and denied 66, while never automatically blocked 31.
  • Intervention burden: 10.9 runtime permission prompts were shown to POLICY participants on average, versus decisions on all 18 actions for HITL.Including rule setup, mean total intervention time was 128.8 s for POLICY and 142.1 s for HITL; the adjusted difference was −12.9 s with 95% CI [−46.3, 20.6].

9 Discussion & Limitations

The discussion attributes POLICY’s weaker protection mainly to ask rules returning decisions to runtime, where users often approved overreach. The authors caution that the simulated setting and coarse categories limit generalization and interpretation.

  • Interpretation: Most of POLICY’s weaker protection appeared in runtime human approvals rather than automatic executions.Of 148 executed overreach actions, 133 were user-approved and 15 ran automatically; POLICY also issued fewer prompts but had higher approval rates.
  • Interpretation: Only 26 of 140 rules settled decisions in advance, while 114 returned authorization to runtime.Ask expressed a preference for case-by-case review without creating an advance boundary.
  • Rule specificity: Every private-data action was overreach, yet 23 of 35 participants chose ask rather than never.The authors note that never would have blocked all private-data overreach and no required action in this study.
  • User experience: High reported control did not imply stronger protection: POLICY participants felt similarly in control despite blocking less overreach.No subjective rating differed reliably between POLICY and either baseline.
  • Design implications: The authors propose previews, narrower rules, and one-time exceptions to make the consequences of ask and broad defaults clearer.These mechanisms aim to support clearer defaults without requiring every action to become a new decision.
  • Limitations: A four-category policy and a scripted online day constrain generalization to long-term use, experienced users, and real decisions involving personal data or money.The authors also note that coarse categories may have contributed to the prevalence of ask, while the design compared complete permission systems rather than isolating authorship.

10 Conclusion

The study finds that user-authored policies did not necessarily strengthen protection: POLICY blocked less overreach than both baselines, while reducing prompts without reliably reducing total intervention time.

  • POLICY blocked less overreach than HITL and AUTO, despite high required-action completion across all three designs.The supporting model estimated odds ratios of 0.349 versus HITL and 0.464 versus AUTO for overreach blocking.
  • POLICY reduced runtime permission prompts relative to HITL, but total intervention time was not reliably lower when setup was included.The prompt-count ratio was 0.605, whereas the intervention-time ratio was 0.931 with a 95% CI including 1.
  • The initial randomized sample produced similar differences, with POLICY blocking 25.0 points less overreach than HITL and 19.0 points less than AUTO.Required-action completion remained high in this sensitivity analysis.

A.4 Exploratory analysis of runtime approval

Runtime approval was highest for overreach actions routed through POLICY prompts, while participants’ earlier preferences did not always carry through to later decisions.

  • Runtime approval: 66.8% of prompted overreach actions were approved in POLICY, compared with 40.3% in HITL and 46.1% in AUTO.The balanced model estimated the same approval rates and found POLICY exceeded HITL by 26.4 percentage points and AUTO by 20.6 points.
  • Interpretation: POLICY’s runtime approval comparison is conditional on actions sent to prompts, so it does not identify a causal effect of POLICY on approval.Only ask-routed actions appear in the POLICY rows.
  • Preference changes: Among 56 overreach actions routed to runtime after a pre-task never response, participants later approved 24, involving 17 of 35 participants.These approvals followed the authored ask rule but differed from the earlier preference.
  • Study procedure: The 18-action simulated day used the same action descriptions across conditions, differing only in whether actions prompted, ran automatically, or were blocked.The task comprised morning, afternoon, and evening scenes.

C Data Validation

Complete records were validated for participant and condition integrity, required action coverage, action order, and consistency of study configuration before exclusions.

  • Validation checks: Records required a valid condition, nonempty participant identifier, completed questionnaires, and action review.These checks were part of the first validation step before preregistered exclusions.
  • Validation checks: Each record was checked for exactly 18 unique actions, including 11 required and 7 overreach actions, with one assigned presentation order.The validation also required all expected fields.
  • Study materials: The study materials documented every participant-facing action disclosure and fixed study label, with “None” marking actions outside the four POLICY categories.These materials are listed in Table 7.
Loading 2608.27443v1…