PaperlayerView exact revision on arXiv

Source-linked AI summary

A Finger on the Scale: Covert Policy Steering through Agentic Skills

Jiarui Li, Jiahao Chen, Chunyi Zhou, Yuwen Pu, Oubo Ma, Zhou Feng, Chunqiang Hu, Shouling Ji

arXiv:2609.02564v1cs.CR

TL;DR

Reusable agent Skills can preserve valid task execution while covertly shifting agent decisions, creating a supply-chain risk that task correctness and output validity alone miss. The paper formalizes Skill Policy Integrity and introduces SkillShift, a constrained black-box framework for covert policy steering; across commerce and dependency selection, it achieves high steering rates while preserving validity, transferring across settings, and evading evaluated scanners.

  • Problem

    Third-party Skills externalize part of an agent’s decision-making logic, enabling undisclosed candidate preferences even when the original task and output interface remain valid.

    Source: Introduction · p. 1

  • Method

    SkillShift is a structured black-box framework that optimizes plausible Skill-level criteria, tie-breaking rules, and semantic examples under task-preservation constraints.

    Source: Introduction · pp. 1–2, Conclusion · p. 7

  • Results

    81% and 63% policy steering rates were achieved across agentic commerce and coding dependency selection, with a 100% valid-output rate, transfer across backends and environments, and most evaluated scanners failing to flag the Skills.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Valid task execution and conventional security checks do not necessarily guarantee Skill Policy Integrity, motivating behavioral auditing with clean–attack comparisons and counterfactual tests.

    Source: Discussion · p. 7, Conclusion · p. 7

    The evaluation is limited to two fixed-candidate domains, three runs per query, detector case studies under specific configurations, and utility analysis relying on an automated judge.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Reusable agent skills extend large language model (LLM) agents with task procedures, tool-use guidance, and output constraints. Yet these skills also act as externalized behavioral policies, which create a supply-chain risk: a third-party skill may preserve the declared task and valid output interface while covertly redirecting agent decisions toward an undisclosed objective. We formalize Skill Policy Integrity, which requires a Skill-induced policy to remain aligned with its declared functionality and the user-authorized objective. We further present SkillShift, a constrained black-box framework for covert policy steering without explicit target command injection or task hijacking. It combines semantically plausible policy edits with hierarchical validation, failure-guided optimization, and strategy compression to preserve effectiveness, output validity, transferability, and inconspicuousness. We instantiate this threat in agentic commerce and software dependency use, with SkillShift achieving attacker-favored selection rates of 81.33% and 63.33% while maintaining a 100% utility-preserving rate. The frozen policies also transfer without further optimization across heterogeneous LLM backends and agent environments. Moreover, the evaluated scanners fail to detect the constructed skills, motivating behavioral auditing of reusable skills as agent policy artifacts.

Introduction

Reusable Skills extend agents’ capabilities while also shaping how they weigh evidence and resolve trade-offs. The paper identifies a supply-chain threat in which manipulated Skills preserve task validity while covertly redirecting decisions, and introduces SkillShift to study it.

  • Reusable Skills encode task procedures, tool guidance, output constraints, and examples, thereby influencing agent decision policies without changing model parameters.Source: Introduction · p. 1
  • Third-party Skills create a supply-chain risk because they delegate part of an agent’s decision-making logic to external providers.Source: Introduction · p. 1
  • Skill Policy Integrity requires Skill-induced behavior to remain faithful to declared functionality and the user-authorized objective, even when task inputs and outputs remain unchanged.Source: Introduction · p. 1
  • SkillShift is a constrained black-box framework that uses plausible policy edits, including framing, tie-breaking, and semantic anchoring, rather than explicit target commands.Source: Introduction · p. 1
  • The work evaluates crosssetting transfer and detection, showing that valid outputs and passing security scans do not guarantee Skill Policy Integrity.Source: Introduction · p. 2

Related Work

Prior work addresses agent attacks through manipulated content, tools, or task environments, but does not explicitly capture persistent selection shifts induced by a shared Skill policy. SkillShift studies this distinct soft-policy attack surface while preserving queries, candidates, and output interfaces.

  • Agent research has studied planning, tool use, indirect prompt injection, instruction–data separation, and trusted-task alignment.Source: LLM Agents and Agent Security · p. 2
  • Prior attacks manipulate webpages, tool names, descriptions, or malicious tool documents to increase selection of attacker-controlled results or tools.Source: Policy and Selection Manipulation · p. 2
  • SkillShift differs by modifying only the shared Skill’s comparison criteria while leaving the user query, candidate set, order, and metadata unchanged.Source: Policy and Selection Manipulation · p. 2
  • Dependency Steering targets hard constraints, whereas SkillShift targets the soft-policy layer through framing, tie-breaking rules, and examples while keeping candidates feasible.Source: Skill Attacks and Detection Gap · p. 3
  • Existing defenses do not explicitly audit persistent selection shifts among legitimate candidates, leaving detection of valid-output policy violations unclear.Source: Skill Attacks and Detection Gap · p. 3

Problem Statement and Threat Model

The threat model considers fixed-candidate agent decisions in which a malicious provider alters only Skill-level comparison guidance to favor a legitimate target. The attacker must increase target selection while preserving task suitability, valid formatting, black-box reusability, and semantic plausibility.

Methodology

SkillShift searches over structured Skill-level strategies that steer candidate comparisons under validity and attack-surface constraints. It combines black-box, failure-guided optimization with hierarchical validation and compression while keeping task inputs, candidates, and model settings fixed.

Attack Overview

Policy-Steering Mechanisms

  • Policy framing changes attribute salience, tie-breaking changes rankings among similarly suitable candidates, and semantic anchoring stabilizes application across queries and backends.Source: Structured Policy Representation · p. 4

Constrained Black-Box Policy Search

Hierarchical Validation

Strategy Update

  • Candidates are generated through local semantic edits and ranked lexicographically by policy steering rate, valid-output rate, and rendered length after a validity threshold.Source: Constrained Black-Box Policy Search · p. 4

Failure-Guided Optimization

  • Failure-guided optimization groups unsuccessful queries by requirements and visible attributes, providing aggregate category feedback rather than instance-specific details.Source: Failure-Guided Coverage Repair · p. 4

Effect-Preserving Strategy Compression

Experiments

Experiments evaluate SkillShift on fixed-candidate shopping and Python dependency-selection tasks, measuring steering, validity, transfer, detection, ablations, and downstream code quality. SkillShift preserves valid outputs and transfers across models and agents, while existing detectors miss it and downstream quality declines under attack.

Setup

  • The evaluation uses fixed candidate sets for sports-product recommendation and Python package selection, with candidate content, attributes, and order unchanged between conditions.The Python task also requires a consistent import statement.Source: Experimental Setup · p. 5

Overall Attack Effectiveness

  • SkillShift achieves 63.33% PSR with 100.00% VR in Python and raises Shopping PSR from 37.33% to 81.33% while maintaining 100.00% VR.The independent all50 evaluation contains 150 model responses per condition.Source: Overall Attack Effectiveness · p. 5

Held-Out Evaluation

  • 100.00% valid-output rates are maintained across Attack conditions, while frozen strategies retain steering on held-out queries.Shopping PSR changes from 85.56% on dev30 to 80.00% on heldout20; Python PSR changes from 46.67% to 75.00%.Source: Generalization to Unseen Queries · p. 6

Transferability

  • Positive Lift appears for every evaluated model–domain combination, with Shopping Lift of 20.00–51.33 percentage points and Python Lift of 52.67–100.00 percentage points.Complete-agent transfer yields positive Lifts of 28.67–94.00 percentage points while maintaining Attack VR of at least 96.67%.Source: Cross-Model and Real-Agent… · p. 6

Ablation Study

  • Removing target semantic cues, task-fit framing, or examples reduces Shopping PSR by 36.00, 30.00, and 28.00 percentage points, while a length-controlled neutral Skill causes a 47.33-point drop.In Python, removing all examples reduces PSR by 33.33 percentage points.Source: Ablation Study · p. 6

Security Detection

  • None of six quantitative detectors distinguishes SkillShift Attack Skills from paired clean Skills, and all three evaluated registries admit or index them without attack-related warnings.Direct-Skill controls receive inconsistent registry treatment, with one registry marking them suspicious, one rejecting them, and one accepting them.Source: Security Detection and Registry… · pp. 6–7

Downstream Utility

  • Despite 100.00% selection-stage VR, SkillShift lowers the mean downstream code-quality score from 0.558 to 0.506.QPR also falls by 16.0 and 20.0 points at thresholds 0.7 and 0.6; the evaluation uses one LLM-judge call per sample without executable tests or significance testing.Source: Impact on Downstream Task Utility · p. 7

Discussion

SkillShift can bias decisions through semantic policy signals while preserving functionally correct outputs, but conventional validity and security checks may miss the behavior. The evaluation is limited in scope and relies partly on automated judgment.

  • SkillShift appears to steer decisions through accumulated semantic signals rather than target-name repetition.Ablations identify semantic cues, task-fit framing, and examples as key contributors, while a length-controlled Skill performs close to the clean condition.Source: Discussion · p. 7
  • Transfer across models suggests functionally correct Skills can remain decision-biased without modifying queries, candidates, model parameters, or output validity.Source: Discussion · p. 7
  • Output validity alone does not guarantee Skill Policy Integrity because existing detectors and registries often fail to identify this behavior.Syntactically valid dependency choices may still reduce downstream quality.Source: Discussion · p. 7
  • Behavioral auditing should use Clean–Attack comparisons and counterfactual tests such as candidate reordering, name replacement, and attribute exchange.Source: Discussion · p. 7
  • The evaluation covers two fixed-candidate domains, three runs per query, and detector case studies under specific configurations, with utility analysis relying on an automated judge.The authors call for broader domains, dynamic candidate sets, stronger statistical evaluation, and behavioral defenses.Source: Discussion · p. 7

Conclusion

SkillShift is a structured black-box framework for covert candidate steering through policy framing, tie-breaking rules, and semantic examples. Across shopping and Python dependency selection, it increases steering while preserving output validity and transferring across settings, showing that valid execution and conventional checks do not ensure Skill Policy Integrity.

  • SkillShift steers candidate selection through policy framing, tie-breaking rules, and semantic examples.Source: Conclusion · p. 7
  • Across shopping recommendation and Python dependency selection, SkillShift increases PSR while preserving high output validity and transferring to unseen queries, multiple models, and complete agent environments.Source: Conclusion · p. 7
  • Valid task execution and conventional security checks do not necessarily guarantee Skill Policy Integrity.The conclusion motivates counterfactual and distribution-based auditing of reusable Skills.Source: Conclusion · p. 7

A. Detailed Experimental Setup

The experiments separate development and held-out evaluation, repeat each query three times, and compare SkillShift with clean, no-Skill, and attack baselines across specified model configurations. Cross-model interpretation emphasizes within-backend Clean–Attack differences because endpoints and implementations may not be immutable or standardized.

Data Splits

Evaluation Batches

Controlled Conditions

Runtime Configuration

  • The Strategy Proposer uses JSON-object responses, while top_p and random seed are not explicitly configured.The openai/ prefix denotes routing through a LiteLLM/OpenAI-compatible interface, not necessarily an official OpenAI endpoint.Source: Models and Runtime Configuration · p. 10
  • Timeouts, retries, and temperature settings are fixed experimental parameters rather than results of systematic hyperparameter search.Source: Models and Runtime Configuration · p. 10

B. Optimization Process and Supplementary Results

Optimization refines a structured attack strategy through staged search and failure refinement, then evaluates the same frozen Skill across model backends. Shopping results show substantial within-backend gains while retaining high validity, with cross-model comparisons focused on Clean–Attack Lift rather than absolute capability.

Shopping Strategy Optimization

  • The initial black-box search reaches 83.33% PSR, failure refinement adds 3.34 percentage points, and compression decreases PSR by 1.11 points while preserving 100.00% VR.The initial structured strategy is an intermediate optimization state rather than an independent baseline.Source: Shopping Strategy Optimization · p. 10

Cross-Model Transfer

Python Dependency Selection

  • For Python dependency selection, absolute PSR values are not used to compare general model capabilities because service-side implementations and maximum output limits are not fully standardized.Interpretation instead uses within-backend Clean–Attack differences.Source: Python Dependency Selection · p. 10

C. Complete Ablation Results

The ablations identify which policy components drive steering and show that compression trades some effectiveness for reduced contextual overhead. Examples and task-oriented criteria matter more than document length alone, while some framing components add limited benefit.

Shopping Recommendation

Python Dependency Selection

Compression

  • The uncompressed strategy outperforms the Lite Strategy by 5.34 percentage points, indicating a moderate effectiveness cost from compression.Source: C. Complete Ablation Results · p. 11

D. Supplementary Overall Attack Results

The supplementary results use Lift as the difference between attack and clean Skill PSR within the same domain and evaluation batch.

E. Full Security-Detector Results

SkillShift Attack Skills evade the evaluated paired detectors, whereas explicit Direct-Skill Injection is more detectable. Detector behavior varies by domain and configuration, and some registry and score interpretations require caution.

SkillShift Detection

  • None of the six evaluated detectors distinguish SkillShift Attack Skills from corresponding Clean Skills in either domain.A Snyk warning on the Python Clean Skill concerns import_statement processing rather than target-dependency manipulation.Source: E. Full Security-Detector Results · p. 11

Detector Scores

Direct-Skill Injection Control

Detector Limitations

Detector Behavior

Registry Evaluation

F. Code-Quality Evaluation Details

The code-quality evaluation uses an automated LLM judge to compare Clean and Attack outputs, while supplementary material documents registry outcomes and the frozen Skills used in evaluation.

Code-Quality Evaluation

Registry Outcomes

Metrics

Limitations

  • GEval results are a single automated LLM-judge assessment, not unit-test correctness or observed execution success.They are used only to compare relative quality between Clean and Attack conditions.Source: F. Code-Quality Evaluation Details · p. 14

Skill Artifacts

Related papers

Loading 2609.02564v1…