PaperlayerView exact revision on arXiv

Source-linked AI summary

A Finger on the Scale: Covert Policy Steering through Agentic Skills

Jiarui Li, Jiahao Chen, Chunyi Zhou, Yuwen Pu, Oubo Ma, Zhou Feng, Chunqiang Hu, Shouling Ji

arXiv:2609.02564v1cs.CR

TL;DR

Reusable agent Skills can preserve valid task execution while covertly shifting agent decisions, creating a supply-chain risk that task correctness and output validity alone miss. The paper formalizes Skill Policy Integrity and introduces SkillShift, a constrained black-box framework for covert policy steering; across commerce and dependency selection, it achieves high steering rates while preserving validity, transferring across settings, and evading evaluated scanners.

  • Problem

    Third-party Skills externalize part of an agent’s decision-making logic, enabling undisclosed candidate preferences even when the original task and output interface remain valid.

  • Method

    SkillShift is a structured black-box framework that optimizes plausible Skill-level criteria, tie-breaking rules, and semantic examples under task-preservation constraints.

  • Results

    81% and 63% policy steering rates were achieved across agentic commerce and coding dependency selection, with a 100% valid-output rate, transfer across backends and environments, and most evaluated scanners failing to flag the Skills.

  • Takeaways & Limitations

    Valid task execution and conventional security checks do not necessarily guarantee Skill Policy Integrity, motivating behavioral auditing with clean–attack comparisons and counterfactual tests.

  • Takeaways & Limitations

    The evaluation is limited to two fixed-candidate domains, three runs per query, detector case studies under specific configurations, and utility analysis relying on an automated judge.

Abstract

from arXiv · show

Reusable agent skills extend large language model (LLM) agents with task procedures, tool-use guidance, and output constraints. Yet these skills also act as externalized behavioral policies, which create a supply-chain risk: a third-party skill may preserve the declared task and valid output interface while covertly redirecting agent decisions toward an undisclosed objective. We formalize Skill Policy Integrity, which requires a Skill-induced policy to remain aligned with its declared functionality and the user-authorized objective. We further present SkillShift, a constrained black-box framework for covert policy steering without explicit target command injection or task hijacking. It combines semantically plausible policy edits with hierarchical validation, failure-guided optimization, and strategy compression to preserve effectiveness, output validity, transferability, and inconspicuousness. We instantiate this threat in agentic commerce and software dependency use, with SkillShift achieving attacker-favored selection rates of 81.33% and 63.33% while maintaining a 100% utility-preserving rate. The frozen policies also transfer without further optimization across heterogeneous LLM backends and agent environments. Moreover, the evaluated scanners fail to detect the constructed skills, motivating behavioral auditing of reusable skills as agent policy artifacts.

Introduction

Reusable Skills extend agents’ capabilities while also shaping how they weigh evidence and resolve trade-offs. The paper identifies a supply-chain threat in which manipulated Skills preserve task validity while covertly redirecting decisions, and introduces SkillShift to study it.

  • Reusable Skills encode task procedures, tool guidance, output constraints, and examples, thereby influencing agent decision policies without changing model parameters.
  • Third-party Skills create a supply-chain risk because they delegate part of an agent’s decision-making logic to external providers.
  • Skill Policy Integrity requires Skill-induced behavior to remain faithful to declared functionality and the user-authorized objective, even when task inputs and outputs remain unchanged.
  • SkillShift is a constrained black-box framework that uses plausible policy edits, including framing, tie-breaking, and semantic anchoring, rather than explicit target commands.
  • The work evaluates crosssetting transfer and detection, showing that valid outputs and passing security scans do not guarantee Skill Policy Integrity.

Related Work

Prior work addresses agent attacks through manipulated content, tools, or task environments, but does not explicitly capture persistent selection shifts induced by a shared Skill policy. SkillShift studies this distinct soft-policy attack surface while preserving queries, candidates, and output interfaces.

  • Agent research has studied planning, tool use, indirect prompt injection, instruction–data separation, and trusted-task alignment.
  • Prior attacks manipulate webpages, tool names, descriptions, or malicious tool documents to increase selection of attacker-controlled results or tools.
  • SkillShift differs by modifying only the shared Skill’s comparison criteria while leaving the user query, candidate set, order, and metadata unchanged.
  • Dependency Steering targets hard constraints, whereas SkillShift targets the soft-policy layer through framing, tie-breaking rules, and examples while keeping candidates feasible.
  • Existing defenses do not explicitly audit persistent selection shifts among legitimate candidates, leaving detection of valid-output policy violations unclear.

Problem Statement and Threat Model

The threat model considers fixed-candidate agent decisions in which a malicious provider alters only Skill-level comparison guidance to favor a legitimate target. The attacker must increase target selection while preserving task suitability, valid formatting, black-box reusability, and semantic plausibility.

  • The setting covers shopping recommendations and Python dependency selection from fixed candidate sets, where Skill-level criteria affect candidate ranking.
  • The attack changes only Skill criteria and examples, leaving queries, candidate data, order, and output interface unchanged.
  • The attacker seeks higher target selection while preserving in-set, task-suitable, correctly formatted outputs and plausible task guidance.
  • The attacker controls one shared Skill and optimizes it from final model outputs without access to parameters, probabilities, gradients, hidden states, or attention.
  • Query-specific rules, candidate identifiers, position shortcuts, and explicit target-selection commands are prohibited, while target names may appear only in permitted examples.

Methodology

SkillShift searches over structured Skill-level strategies that steer candidate comparisons under validity and attack-surface constraints. It combines black-box, failure-guided optimization with hierarchical validation and compression while keeping task inputs, candidates, and model settings fixed.

  • Attack Overview: The strategy must create persistent preference without explicit commands, preserve the task boundary and interface, and generalize from sparse black-box feedback.
  • Attack Overview: SkillShift represents strategies as global principles, task rules, tie-breakers, and semantic examples, then appends them deterministically to the base Skill.
  • Policy-Steering Mechanisms: Policy framing changes attribute salience, tie-breaking changes rankings among similarly suitable candidates, and semantic anchoring stabilizes application across queries and backends.
  • Constrained Black-Box Policy Search: The search uses final victim responses on a fixed development set, with no access to logits, gradients, hidden states, or attention values.
  • Hierarchical Validation: Hierarchical validation checks schema, surface isolation, and content constraints before candidate execution, recording rejected violation types.
  • Strategy Update: Candidates are generated through local semantic edits and ranked lexicographically by policy steering rate, valid-output rate, and rendered length after a validity threshold.
  • Failure-Guided Optimization: Failure-guided optimization groups unsuccessful queries by requirements and visible attributes, providing aggregate category feedback rather than instance-specific details.
  • Effect-Preserving Strategy Compression: Compression removes, merges, shortens, and generalizes strategy content while constraining valid-output rate and limiting policy steering-rate degradation.

Experiments

Experiments evaluate SkillShift on fixed-candidate shopping and Python dependency-selection tasks, measuring steering, validity, transfer, detection, ablations, and downstream code quality. SkillShift preserves valid outputs and transfers across models and agents, while existing detectors miss it and downstream quality declines under attack.

  • Setup: The evaluation uses fixed candidate sets for sports-product recommendation and Python package selection, with candidate content, attributes, and order unchanged between conditions.The Python task also requires a consistent import statement.
  • Overall Attack Effectiveness: SkillShift achieves 63.33% PSR with 100.00% VR in Python and raises Shopping PSR from 37.33% to 81.33% while maintaining 100.00% VR.The independent all50 evaluation contains 150 model responses per condition.
  • Held-Out Evaluation: 100.00% valid-output rates are maintained across Attack conditions, while frozen strategies retain steering on held-out queries.Shopping PSR changes from 85.56% on dev30 to 80.00% on heldout20; Python PSR changes from 46.67% to 75.00%.
  • Transferability: Positive Lift appears for every evaluated model–domain combination, with Shopping Lift of 20.00–51.33 percentage points and Python Lift of 52.67–100.00 percentage points.Complete-agent transfer yields positive Lifts of 28.67–94.00 percentage points while maintaining Attack VR of at least 96.67%.
  • Ablation Study: Removing target semantic cues, task-fit framing, or examples reduces Shopping PSR by 36.00, 30.00, and 28.00 percentage points, while a length-controlled neutral Skill causes a 47.33-point drop.In Python, removing all examples reduces PSR by 33.33 percentage points.
  • Security Detection: None of six quantitative detectors distinguishes SkillShift Attack Skills from paired clean Skills, and all three evaluated registries admit or index them without attack-related warnings.Direct-Skill controls receive inconsistent registry treatment, with one registry marking them suspicious, one rejecting them, and one accepting them.
  • Downstream Utility: Despite 100.00% selection-stage VR, SkillShift lowers the mean downstream code-quality score from 0.558 to 0.506.QPR also falls by 16.0 and 20.0 points at thresholds 0.7 and 0.6; the evaluation uses one LLM-judge call per sample without executable tests or significance testing.

Discussion

SkillShift can bias decisions through semantic policy signals while preserving functionally correct outputs, but conventional validity and security checks may miss the behavior. The evaluation is limited in scope and relies partly on automated judgment.

  • SkillShift appears to steer decisions through accumulated semantic signals rather than target-name repetition.Ablations identify semantic cues, task-fit framing, and examples as key contributors, while a length-controlled Skill performs close to the clean condition.
  • Transfer across models suggests functionally correct Skills can remain decision-biased without modifying queries, candidates, model parameters, or output validity.
  • Output validity alone does not guarantee Skill Policy Integrity because existing detectors and registries often fail to identify this behavior.Syntactically valid dependency choices may still reduce downstream quality.
  • Behavioral auditing should use Clean–Attack comparisons and counterfactual tests such as candidate reordering, name replacement, and attribute exchange.
  • The evaluation covers two fixed-candidate domains, three runs per query, and detector case studies under specific configurations, with utility analysis relying on an automated judge.The authors call for broader domains, dynamic candidate sets, stronger statistical evaluation, and behavioral defenses.

Conclusion

SkillShift is a structured black-box framework for covert candidate steering through policy framing, tie-breaking rules, and semantic examples. Across shopping and Python dependency selection, it increases steering while preserving output validity and transferring across settings, showing that valid execution and conventional checks do not ensure Skill Policy Integrity.

  • SkillShift steers candidate selection through policy framing, tie-breaking rules, and semantic examples.
  • Across shopping recommendation and Python dependency selection, SkillShift increases PSR while preserving high output validity and transferring to unseen queries, multiple models, and complete agent environments.
  • Valid task execution and conventional security checks do not necessarily guarantee Skill Policy Integrity.The conclusion motivates counterfactual and distribution-based auditing of reusable Skills.

A. Detailed Experimental Setup

The experiments separate development and held-out evaluation, repeat each query three times, and compare SkillShift with clean, no-Skill, and attack baselines across specified model configurations. Cross-model interpretation emphasizes within-backend Clean–Attack differences because endpoints and implementations may not be immutable or standardized.

  • Data Splits: Each domain contains 50 queries, split into dev30 for optimization and heldout20 for final evaluation; held-out queries remain unused until strategy freezing.
  • Evaluation Batches: Each query is executed three times, yielding 90 dev30, 60 heldout20, and 150 all50 responses, without treating repeated runs as independent query instances.
  • Evaluation Batches: Split-specific evaluation separates dev30 and heldout20, whereas an independent all50 rerun executes all 50 queries in a new batch.
  • Evaluation Batches: Clean–Attack Lift is computed only within the same evaluation batch, and hit counts are not aggregated across independently executed batches.
  • Controlled Conditions: Clean and Attack conditions use identical queries, candidate sets, attributes, and orders; cached candidate records are read directly for formal evaluation.
  • Runtime Configuration: The Strategy Proposer uses JSON-object responses, while top_p and random seed are not explicitly configured.The openai/ prefix denotes routing through a LiteLLM/OpenAI-compatible interface, not necessarily an official OpenAI endpoint.
  • Runtime Configuration: Timeouts, retries, and temperature settings are fixed experimental parameters rather than results of systematic hyperparameter search.

B. Optimization Process and Supplementary Results

Optimization refines a structured attack strategy through staged search and failure refinement, then evaluates the same frozen Skill across model backends. Shopping results show substantial within-backend gains while retaining high validity, with cross-model comparisons focused on Clean–Attack Lift rather than absolute capability.

  • Shopping Strategy Optimization: The initial black-box search reaches 83.33% PSR, failure refinement adds 3.34 percentage points, and compression decreases PSR by 1.11 points while preserving 100.00% VR.The initial structured strategy is an intermediate optimization state rather than an independent baseline.
  • Cross-Model Transfer: The same frozen Attack Skill is directly loaded into every model backend without model-specific strategy optimization.
  • Cross-Model Transfer: Across all six model backends, Shopping PSR Lift ranges from 20.00 to 51.33 percentage points while Attack VR remains above 93%.
  • Python Dependency Selection: For Python dependency selection, absolute PSR values are not used to compare general model capabilities because service-side implementations and maximum output limits are not fully standardized.Interpretation instead uses within-backend Clean–Attack differences.

C. Complete Ablation Results

The ablations identify which policy components drive steering and show that compression trades some effectiveness for reduced contextual overhead. Examples and task-oriented criteria matter more than document length alone, while some framing components add limited benefit.

  • Shopping Recommendation: Removing the target semantic cue, task-fit rubric, or all examples substantially decreases PSR.
  • Shopping Recommendation: The length-controlled neutral Skill remains close to Clean PSR, indicating that attack effectiveness is not explained solely by increased document length.
  • Shopping Recommendation: Removing professional framing or tie-breaking rules causes only small PSR decreases in the optimized strategy.
  • Python Dependency Selection: Removing all Python examples causes the largest decrease, while usage descriptions and few-shot examples individually have smaller effects.
  • Compression: The uncompressed strategy outperforms the Lite Strategy by 5.34 percentage points, indicating a moderate effectiveness cost from compression.

D. Supplementary Overall Attack Results

The supplementary results use Lift as the difference between attack and clean Skill PSR within the same domain and evaluation batch.

  • Lift is computed relative to the Clean Skill in the same domain as an absolute difference in percentage points.

E. Full Security-Detector Results

SkillShift Attack Skills evade the evaluated paired detectors, whereas explicit Direct-Skill Injection is more detectable. Detector behavior varies by domain and configuration, and some registry and score interpretations require caution.

  • SkillShift Detection: None of the six evaluated detectors distinguish SkillShift Attack Skills from corresponding Clean Skills in either domain.A Snyk warning on the Python Clean Skill concerns import_statement processing rather than target-dependency manipulation.
  • Detector Scores: ProtectAI’s Shopping attack score rises from 7.32 × 10−5 to 0.3052 but remains below the 0.5 benign threshold.In Python, both Clean and Attack scores remain close to zero.
  • Direct-Skill Injection Control: Direct-Skill Injection is detected by four of six detectors in each domain, corresponding to a 66.7% detection rate.
  • Detector Limitations: The Shopping STARS result is inconclusive because its English rule triggers may not cover the Chinese-language request.
  • Detector Behavior: Existing detectors are substantially more effective at identifying overt instruction-override patterns than SkillShift’s implicit policy steering.
  • Registry Evaluation: The registry evaluation covers ClawHub, Tencent SkillHub, and vskill security pipelines for paired clean, attack, and control artifacts.

F. Code-Quality Evaluation Details

The code-quality evaluation uses an automated LLM judge to compare Clean and Attack outputs, while supplementary material documents registry outcomes and the frozen Skills used in evaluation.

  • Code-Quality Evaluation: The downstream experiment evaluates 50 Clean outputs and 50 Attack outputs using DeepEval’s GEval Code Quality metric.
  • Code-Quality Evaluation: The judge assesses task matching, dependency use, implementation completeness, and apparent API misuse or other quality problems.
  • Registry Outcomes: All six artifacts passed vskill Tier 1 and Tier 2 checks at capture time, while Tencent SkillHub blocked both Direct-Skill Injection artifacts before publication.
  • Metrics: QPR@0.7 uses the primary DeepEval threshold, whereas QPR@0.6 is computed retrospectively from saved raw scores without a second judge call.
  • Limitations: GEval results are a single automated LLM-judge assessment, not unit-test correctness or observed execution success.They are used only to compare relative quality between Clean and Attack conditions.
  • Skill Artifacts: The frozen SkillShift Attack Skill preserves the task definition, candidate constraints, and output interface while appending task-fit guidance, criteria, tie-breaking rules, and examples.
Loading 2609.02564v1…