PaperlayerView exact revision on arXiv

Source-linked AI summary

PyCache Trap: The Inspection-Execution Gap in Agent Skill Scanners

Jie Liao, Simeng Qin, Wenqi Ren, Wei Zhou, Junhao Wen, Ranjie Duan, Yang Liu, Xiaojun Jia

arXiv:2610.10612v1cs.CRcs.SE

TL;DR

Scanners inspect documentation and visible source, but Python may execute a different bundled cache body. PyCache Trap measures this gap, while EAV validates runtime-selected artifacts through a typed execution graph and trusted reproduction. Across the evaluations, PyCache Trap succeeds against scanners and EAV detects source-present substitutions and performs across attack families.

  • Problem

    Current scanners inspect documentation and source, but Python may load a compiled cache whose executable body was not derived from the inspected source (Python Software Foundation, n.d.; Peterson, 2017).

    Source: §1 · p. 1

  • Method

    PyCache Trap links a loader-accepted substituted cache to a task-relevant invocation and adapts visible wording, while EAV connects inspected content to runtime artifacts and validates provenance through trusted reproduction.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across seven scanners, PyCache Trap achieves 94–100% ASR on 100 Skills; EAV detects all 100 evaluated source-present substitutions and reaches 92.8% Recall at 10.0% FPR in the mixed-family evaluation.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Admission should account for artifacts the configured runtime can select, checking reachable caches against trusted reproductions of validated source.

    Source: §1 · p. 2, §3.3 · p. 5

    The threat model assumes the publisher controls package contents but not the scanner, agent model, interpreter, or sandbox, with visible source and a compatible cache in the same package.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Agent skills combine instructions with executable resources, giving third-party packages access to an agent's runtime. Existing skill scanners inspect documentation and visible source, but Python may execute a bundled bytecode cache with different behavior. We study this gap between inspection and execution through PyCache Trap, which pairs benign source with a substituted cache accepted by the loader and connects it to a task-relevant invocation. Scanner-guided rewriting changes the invocation wording while preserving the cache body, separating package admission from recognition of the concealed behavior. Across 100 skills and seven scanners, PyCache Trap achieves 94-100% attack success, with no semantic recognition of the cache-resident behavior. We propose execution-aware validation (EAV) to connect inspected instructions, scripts, imports, and runtime artifacts in a typed execution graph. EAV combines grounded behavioral analysis with trusted reproduction of compiled artifacts. It detects all 100 evaluated source-present cache substitutions and reaches 92.8% Recall at 10.0% FPR across five attack families and 200 benign skills. The results support checking the executable artifacts a runtime can select as part of skill admission, within the supported loaders and code-object normalization. The code is released at https://github.com/leo0481/PyCacheTrap.

1 INTRODUCTION

The introduction frames a gap between scanner-visible source and the executable cache Python may select, then presents PyCache Trap to measure it and EAV to validate runtime-selected artifacts.

1 INTRODUCTION

  • Python may load a compiled cache rather than compile visible source, so scanners can approve source while authorizing a different executable body (Python Software Foundation, n.d.; Peterson, 2017).Cache-invalidation metadata determines cache eligibility but does not establish that the executable body was derived from inspected source.Source: §1 · p. 1
  • PyCache Trap separates package admission from semantic recognition by retaining assigned cache bytes while an LLM rewriter revises scanner-visible invocation wording under scanner feedback.The evaluation reports blocking and semantic recognition before and after adaptation.Source: §1 · p. 2
  • EAV connects instructions, scripts, imports, and runtime artifacts in a typed execution graph, grounding behavioral analysis and checking reachable caches against trusted reproductions of validated source.The paper characterizes this as a response to the inspection–execution gap in scanner-mediated Skill workflows.Source: §1 · pp. 1–2
  • 94–100% ASR across seven scanners is achieved by PyCache Trap on 100 Skills, with scanner-facing wording adapted while cache bytes remain fixed.The construction connects a loader-accepted substituted cache to a task-relevant invocation.Source: §1 · p. 2
  • 100 evaluated source-present substitutions missed by header consistency are detected by EAV, which achieves 92.8% Recall at 10.0% FPR in the mixed-family evaluation.EAV validates runtime-selectable artifacts through a typed execution graph and reachability, forbidden-composition, and provenance-closure criteria.Source: §1 · p. 2

2 RELATED WORK

Related work covers security risks in Skills and agent workflows, scanning and provenance defenses, and the executable-artifact gap addressed by this paper.

2 RELATED WORK

  • Prior Skill studies report repository vulnerabilities, prompt-injection attacks, and optimization of malicious instructions, alongside broader agent-security evaluations of untrusted inputs across planning and tool execution (Liu et al., 2026b; Schmotz et al., 2026; Jia et al., 2026a; Liu et al., 2026a; Zhang et al., 2025; Ling et al., 2026; Kolluri et al., 2026).Skills package instructions, scripts, and resources for on-demand agent use.Source: §2 · pp. 2–3
  • The paper complements prior scanning and provenance work by analyzing the executable artifact selected after Skill instructions lead to an invocation; recovering a visible invocation does not validate its bytecode body.Python’s import rules and PEP 552 govern cache selection and invalidation but do not establish executable provenance (Python Software Foundation, n.d.; Peterson, 2017).Source: §2 · p. 3
  • Existing defenses include pattern matching, behavioral analysis, permission review, static and semantic auditing, capability verification, and multimodal analysis with execution simulation (Cisco AI Defense, n.d.; Snyk, n.d.; fedrov2025, n.d.; Liu et al., 2026b; Ying et al., 2026; Bhardwaj, 2026; Jia et al., 2026b).Package-level static and dynamic analysis and artifact-production systems such as in-toto are also discussed (DAS Lab, n.d.; Torres-Arias et al., 2019).Source: §2 · p. 3

3 METHOD

The method formalizes the gap between what a scanner inspects and what a Python runtime executes, then evaluates that gap with PyCache Trap and addresses it with execution-aware validation (EAV). PyCache Trap holds the cache body fixed while varying scanner-facing wording; EAV links inspected content to runtime-selected artifacts and checks their behavior and provenance.

3.1 THREAT MODEL AND PROBLEM FORMULATION

  • A gap arises when admission based on a scanner’s view authorizes an executable representation the scanner did not validate.The threat model places visible source and a compatible bundled cache in one Skill, with a relevant task invoking local programs; cache acceptance does not prove that its body matches trusted compilation of the source.Source: §3.1 · p. 3

3.2 PYCACHE TRAP: EVALUATING THE SOURCE–CACHE MISMATCH

  • PyCache Trap pairs inspectable source with a loader-compatible cache containing different behavior, then connects the module to a task-relevant invocation.It uses a normally compiled cache from the same source as a negative control; 13 of 100 Skills support placing the invocation on an existing script path, and task relevance does not establish preservation of original utility.Source: §3.2 · p. 4
  • Scanner-conditioned rewriting changes the Skill description in response to findings while preserving the invoked command and executed cache body.The search varies invocation placement and wording, retaining the source/cache pair, loader header, lazy import, and entry naming; it stops on acceptance, exhaustion of invocation nodes, or the candidate budget.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3 EAV: EXECUTION-AWARE SKILL VALIDATION

  • EAV constructs a typed execution graph connecting instruction phases, scripts, source modules, caches, and effects under the configured runtime.Invocation extraction, AST and loader analysis resolve explicit calls, unambiguous imports, and cache selection; unresolved dynamic targets remain review items.Source: §3.3 · p. 5
  • EAV projects prose, code, images, and bytecode onto shared effects, retaining model-proposed effects only when grounded in inspected evidence.Deterministic extraction also covers commands, reversible encodings, OCR-recovered commands, and bytecode constants and operations; partially grounded effects remain advisory and ungrounded proposals are discarded.Source: §3.3 · p. 5
  • EAV checks reachability, forbidden effect compositions, and provenance closure, requiring reachable caches to match trusted reproduction of paired inspected source.Trusted reproduction uses an interpreter-matched compilation and normalized executable fields and typed constants; unsupported pairings yield abstention. Classification then uses configured policy and context, while purpose review cannot override high-confidence closure evidence.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate PyCache Trap against existing scanners and assess EAV’s artifact validation and broader defense performance. PyCache Trap remains highly successful across scanners, while EAV detects source-present substitutions and performs strongly across five attack families.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses 100 clean-base Skills, seven scanners, five candidate iterations, and 200 benign Skills for FPR measurement.The 500 adversarial candidates comprise PyCache Trap and four text/image variants; 100 legitimate-cache fixtures test compatibility.Source: §4.1 · pp. 6–7

4.2 ATTACK EFFECTIVENESS

  • 94–100% ASR across seven scanners, including 94% against ExecScan, with zero semantic recognition of the concealed cache behavior.EAV reduces PyCache Trap ASR to 0% through deterministic provenance validation.Source: Table 1 · p. 7, §4.2 · p. 7
  • EAV records 0% ASR in every measured baseline-source transfer cell, while baseline-source ASR remains 92–100% against Cisco, ExecScan, and SkillFortify.Adaptive runtime confirmation finds admission plus a runtime-signature match for 76–83 of 100 victims per evaluated scanner.Source: §4.2 · p. 8, Figure 5 · p. 8, Table 2 · p. 8

4.3 DEFENSE EFFECTIVENESS

  • Trusted reproduction and integrated EAV detect all 100 source-present substitutions; direct bytecode semantics also detects all but flags 17 of 100 legitimate caches.Trusted reproduction accepts 86 compatible source/cache pairs and abstains on 14 sourceless fixtures; deployed EAV rejects those 14, a 14% rejection rate across the legitimate-cache cohort.Source: §4.3 · p. 8, Table 3 · p. 9
  • 92.8% Recall, 95.9% Precision, 94.3 F1, and 10.0% FPR: EAV leads the defense comparison over five attack families and benign Skills.ExecScan reaches 70.0% Recall, SkillFortify 8.6% Recall, and HSS over-flags benign inputs.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 ABLATION AND FURTHER ANALYSIS

  • Removing reachability lowers Recall from 93.0% to 79.0%, while removing closure, composition, or context changes it by at most 0.2 points.Removing bytecode findings lowers FPR from 8.5% to 6.0% without reducing Recall; rebuilding caches from inspected source removes substituted bytes in all 99 code-bearing packages.Source: §4.4 · p. 9
  • Trusted reproduction detects every namespace and dynamic-import substitution with paired source, but the graph abstains on all 50 sourceless custom-hook cases.The loaded artifacts in those custom-hook cases remain unresolved in the graph.Source: §4.4 · p. 9
  • On the 556-Skill MalSkillBench subset, EAV reaches 96.0% Precision, 97.9% Recall, 96.9% F1, and 9.6% FPR.On the 100-package substitution set, integrated EAV latency is 39.82/190.93 ms median/p95; on the mixed workload it uses 15.6 s and 10,214 tokens per Skill.Source: §4.4 · p. 9

5 CONCLUSION

PyCache Trap exposes a gap between inspected source and Python-selected bytecode, while EAV connects inspected instructions to runtime artifacts and checks executable provenance. The conclusion reports strong scanner evasion and defense results within the evaluated settings.

5 CONCLUSION

  • 94–100% ASR across seven scanners, with no semantic recognition of the concealed cache behavior after invocation wording changed while cache bytes stayed fixed.The result shows that approving visible source does not establish the provenance of code executed during a Skill invocation.Source: §5 · p. 10
  • EAV connects inspected instructions to runtime artifacts, using trusted reproduction for executable provenance and grounded behavioral analysis for text and image carriers.The language-model layer is bounded to extracting prose-level behavior; trusted reproduction applies where the evaluated loader and code-object normalization apply.Source: §5 · p. 10
  • The evaluated results support making the relationship between inspected content and runtime-selected code an explicit part of Skill admission.The stated scope is the evaluated settings and supported loader and normalization behavior.Source: §5 · p. 10

ETHICS STATEMENT

The work studies the inspection–execution gap to inform stronger skill-vetting defenses while acknowledging dual-use risks. Its experiments use isolated environments and avoid real-world deployment.

ETHICS STATEMENT

  • The characterized cache-substitution techniques carry dual-use risks, including possible unauthorized data access, privilege escalation, file modification, or backdoor execution.The statement describes potential misuse, not harms demonstrated by the experiments.Source: ETHICS STATEMENT · p. 11
  • Experiments used isolated Docker environments, controlled tasks, dummy secrets, and synthetic target behaviors, without attacking production services or third parties.No manipulated skill was deployed to a public platform.Source: ETHICS STATEMENT · p. 11
  • To reduce misuse, the release omits directly reusable malicious payloads and end-to-end attack scripts, while including the defense, evaluated scanners, benign corpora, and sealed result data.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The reproducibility statement points to supplementary materials and appendices for code, protocols, measurements, implementation details, and prompts. Matching the evaluated model backends, permissions, and runtime settings is required to reproduce results.

REPRODUCIBILITY STATEMENT

  • The paper's source code is provided as a sanitized anonymous repository in the supplementary materials.Source: REPRODUCIBILITY STATEMENT · p. 11
  • Sections 3 and 4.1–4.3 cover the threat model, attack constraints, evaluated platforms and metrics, and attack and defense evaluations.Source: REPRODUCIBILITY STATEMENT · p. 11
  • Reproducing the reported results requires matching the evaluated model backends, tool permissions, and runtime settings.Appendices provide the scanner protocol, measurement and defense details, loader variants, and language-model prompts.Source: REPRODUCIBILITY STATEMENT · p. 11

AI USAGE STATEMENT

The statement distinguishes manuscript-preparation uses of generative AI from the language-model procedures evaluated in the paper. The accompanying protocol and results material describe candidate evaluation and the measurements supporting Section 4.

AI USAGE STATEMENT

  • Generative AI was used for language polishing and manuscript preparation, not to develop research ideas, design experiments, implement research code, or analyze results.Reported experimental uses were the evaluated scanners, the defense's grounded extraction layer, and the attack's invocation-wording adaptation component.Source: AI USAGE STATEMENT · p. 11

A DETAILED VALIDATION PROTOCOL

  • Each candidate was evaluated once by its target scanner; scanner ASR counts admitted packages, while semantic recognition requires a finding identifying the concealed cache body.The runtime check used a Claude Code harness with MiniMax-M3 and one agent trial per candidate; its joint outcome required admission and a runtime-signature match.Source: A DETAILED VALIDATION PROTOCOL · p. 14

B DETAILED RESULTS

  • The detailed-results section supplies experimental units, transfer results, artifact-validation measurements, and criterion ablations supporting Section 4.Source: B DETAILED RESULTS · p. 14

B.2 SCANNER ATTACK-SUCCESS RESULTS

Baseline scanners retain high attack success while semantic recognition of the concealed behavior remains zero. EAV is evaluated across attack families, criterion ablations, and loader configurations.

B.2 SCANNER ATTACK-SUCCESS RESULTS

  • Baseline scanners retain 92–100% ASR for Cisco, ExecScan, and SkillFortify, while HSS ranges from 10% to 62%; semantic recognition remains zero.The three Claude-based target scanners admit all baseline-source candidates.Source: §B.2 · p. 14

B.3 ARTIFACT VALIDATION, COMPATIBILITY, AND COST

  • The validation matrix includes 100 source-present substituted-cache packages and 100 legitimate-cache fixtures across seven compatibility categories.Loader behavior was self-validated in a Python 3.11-matched container.Source: §B.3 · p. 14
  • 100.0% Recall for PyCache Trap, 94.0% for SKILLJECT, 91.0% for SKILLCAMO-Cloze, 90.0% for SKILLCAMO-Split, and 89.0% for SKILLCAMO.The 464 detections yield 92.8% aggregate Recall; provenance validation detects cache substitutions.Source: §B.3 · p. 14

B.4 CRITERION ABLATION

  • Reachability contributes the largest Recall gain, while bytecode findings add 2.5 percentage points of FPR and detect no attack alone.The FPR contribution comes from five benign Skills whose setup-script documentation triggers a mandatory-execution prose signal; the criterion is retained for sourceless-bytecode provenance when trusted reproduction abstains.Source: §B.4 · p. 15

B.5 LOADER-SHAPE COVERAGE

  • Trusted reproduction detects all 50 namespace-package and 50 dynamic-import substitutions, but abstains on all 50 sourceless custom-hook fixtures.The graph flags every dynamic-import case but leaves the loaded custom-hook artifact unresolved.Source: §B.5 · p. 15

C IMPLEMENTATION DETAILS

EAV links inspected instructions and code to runtime-loadable artifacts, then applies provenance checks, deployment-owned endpoint policy, and contextual classification. Its semantic-recognition rubric requires findings to identify the concealed behavior or the artifact carrying it.

C IMPLEMENTATION DETAILS

  • Deterministic extractors resolve script references, imports, and supported loader paths, while grounded language-model passes recover instruction phases and prose-induced actions.The graph traces invocations to runtime-loadable artifacts and propagates effects through file and environment channels; computed targets and unsupported paths remain unresolved.Source: C IMPLEMENTATION DETAILS · p. 15
  • For supported source/cache pairs, EAV recompiles inspected source in a matched interpreter and recursively compares normalized code-object structure and constants.Unsupported pairings and source-compilation failures remain unresolved rather than evidence of derivation; filenames and line-location metadata are excluded from comparison.Source: C IMPLEMENTATION DETAILS · pp. 15–16
  • Retrieval authorization requires exact scheme, canonical host, and effective port against scanner-owned allowlists; data-release destinations require separate authorization and default to none.Package metadata cannot populate either list, and download–execute compositions retain findings even for allowed retrieval origins.Source: C IMPLEMENTATION DETAILS · p. 16, Table S4 · p. 16
  • The endpoint policy is static; enforcing destinations across redirects, DNS changes, or unmodeled runtime behavior requires deployment egress control.The scanner logs the effective origins for each run.Source: C IMPLEMENTATION DETAILS · p. 16
  • Stage IV classifies each Skill using contextualized, weighted findings and a fixed 0.55 risk cutoff, while unresolved artifacts remain review annotations.Grounded provenance violations retain a harmful verdict; purpose review can dismiss heuristic findings associated with legitimate Skill functions.Source: C IMPLEMENTATION DETAILS · p. 16

C.1 SEMANTIC-RECOGNITION RUBRIC

  • Semantic recognition is scored only when stored findings identify the substituted cache artifact, concealed behavior, or carrying file; merely mentioning a cache is negative.The unit is the stored finding set for one scanner–Skill evaluation.Source: §C.1 · p. 16
  • The rubric yields zero positive semantic-recognition labels across seven baseline scanners, although a generic warning can still block admission.Source: §C.1 · p. 16

D LIMITATIONS

This section documents EAV’s evidence boundaries and deployment-owned endpoint controls, and describes the prompts used to adapt PyCache Trap documentation and construct execution graphs. It explains how recovered effects across source, prose, and runtime-facing representations are reviewed against declared purpose, with explicit authorization required for capabilities and benign verdicts limited to plainly declared functions without specified malicious behaviors.

D LIMITATIONS

  • EAV’s evidence is bounded to fixed cohorts, scanner configurations, runtimes, and measured single-scan evaluations, while unsupported loaders and caches without inspected source are held for review.Model-assisted recovery grounds findings in source evidence, keeping admission conservative when evidence is absent.Source: D LIMITATIONS · p. 17
  • Endpoint authorization remains deployment-owned: dependency origins authorize retrieval, data release needs a separate grant, and package declarations extend neither authorization list.Origin-level policy composes with deployment-side egress controls for redirect and DNS enforcement.Source: D LIMITATIONS · p. 17

E ATTACK AND DEFENSE PROMPTS

  • The attack-and-defense prompt templates record source files, symbols, and SHA-256 hashes in a manifest, identify optional passes in configuration, and use deterministic cache construction and trusted reproduction.They support documentation adaptation and the graph construction, effect projection, and context review stages.Source: E ATTACK AND DEFENSE PROMPTS · p. 17

E.1 PYCACHE TRAP: DOCUMENTATION ADAPTATION

  • Documentation-adaptation prompts add a domain-framed initialization reference and revise its wording or placement using scanner feedback while keeping the cache body fixed.The initial edit preserves the skill’s primary purpose; the feedback revision aims to make the document less suspicious without changing that purpose.Source: §E.1 · pp. 17, 19

E.3 EAV: EXECUTION GRAPH AND EFFECT RECOVERY

  • Execution-graph prompts partition files into phases, identify cross-file calls and data channels, and project concrete actions—including those implied by documented steps—into effects.Inputs include line-numbered file chunks and declared skill intent; phases follow logical steps and add only real control or data edges.Source: §E.3 · pp. 21–22
  • EAV traces data across files and transforms, requiring verbatim source anchors for effects and carrier names at both ends of semantic-taint edges.The method targets flows such as a credential read in one script and sent over a network call in another.Source: §E.3 · p. 23
  • EAV treats prose that induces sensitive actions—including credential reads, external transmission, persistence, or agent-control decisions—as reportable effects even without commands.Soft-framed checks such as verifying credential availability still count as access.Source: §E.3 · pp. 23–24
  • EAV projects indirect external destinations and abstract credential-access instructions into literal targets, grounding each finding in a source line.External URL or IP definitions count as egress targets even without a send verb; credential-store checks can resolve to concrete paths.Source: §E.3 · p. 25

E.4 EAV: SCOPE AND PURPOSE REVIEW

  • Stage IV compares package-authored intent with observed effects and flags only effects that contradict or are plainly unrelated to the declared purpose.Normal implied operations, such as using a credential for an independently authorized service, remain in scope.Source: §E.4 · p. 26
  • Endpoint authorization comes only from deployment-owned trusted origins; a skill’s declaration cannot authorize a destination, and unlisted endpoints require data-flow assessment.Provenance findings retain deterministic verdicts that the scope and purpose reviews cannot override.Source: §E.4 · p. 26
  • Capabilities belong in the authorization record only when the skill explicitly authorizes them; under-specification is neither authorization nor a violation, so omit uncertain entries.The record includes the capability and a verbatim substring of the authorizing declaration.Source: §E.4 · p. 27
  • Auditors must read every flagged file before deciding, because concealed payloads may reside in quietly run underscore-prefixed helpers.Examples include _cleanup.py, _reset_state.py, and _api_config.py.Source: §E.4 · p. 27
  • Treat deletion or overwriting, credential or secret access, and unauthorized network contact or fetch-then-run behavior as malicious, even when framed as setup or cleanup.Package-authored endpoint declarations do not authorize network destinations.Source: §E.4 · p. 27
  • Treat decoded-blob execution, unrelated shell or subprocess activity, and coercion to run a bundled script first or without asking as malicious.The review says never to rescue such findings even when framed as cache, initialization, or setup.Source: §E.4 · p. 27
  • A benign verdict is reserved for plainly declared functions without the listed behaviors; when in doubt, answer MALICIOUS.Examples include running the skill’s analysis script on user data, adding its CLI to PATH, or using a declared tool rather than guessing.Source: §E.4 · p. 27

E.2 EAV: SHARED ANALYSIS BOUNDARY

EAV’s text-analysis boundary adds a shared guard and wraps Skill content as text to analyze. Model outputs then pass through grounding before contributing to a binary decision.

E.2 EAV: SHARED ANALYSIS BOUNDARY

  • Text-analysis calls prepend a shared security guard and wrap the Skill text in a marked input envelope.The shared prefix instructs the model to treat the enclosed Skill as material for analysis, not as a request to follow.Source: §E.2 · pp. 20–21
  • Outputs enter the grounding step in Eq. 5 before contributing to the binary decision in Eq. 7.Source: §E.2 · p. 20
Loading 2610.10612v1…