PaperlayerView exact revision on arXiv

Source-linked AI summary

Scan the Skill, Govern the Action: Composing Registry Verdicts with Runtime Consequence Control

Rohit Taneja, Travis Weber

arXiv:2609.12001v1cs.CRcs.SE

TL;DR

Publish-time scanners assess whether a skill is malicious, but action permission depends on the operator’s policy and context. The paper measures mismatches between skill documentation and runtime actions, and proposes a deterministic resolver with a trust ledger. In its live measurements, 34.7% of commands had a consequence class absent from the document, and the gate stopped all 23 attempted forbidden actions across 53 skills.

  • Problem

    Publish-time screening evaluates maliciousness, while whether an action is permitted depends on the operator’s policy and facts unavailable when a skill is published.

    Source: §1.1 · p. 3

  • Method

    The paper proposes a deterministic resolver for actions paired with a per-lane trust ledger and autonomy thresholds derived from operator risk tolerance.

    Source: §11 · p. 19

  • Results

    34.7% of commands carried a consequence class absent from the skill document, and the gate held or blocked all 23 attempted forbidden actions across 53 cleared skills.

    Source: §11 · p. 19

  • Takeaways & Limitations

    Action permission is a policy-dependent property distinct from artifact maliciousness, so a publish-time verdict does not answer whether an action is permitted for an operator.

    Source: §1.1 · p. 3

    The trials are neither independent nor sampled from the population being licensed, and they bound failure under human review rather than without it.

    Source: §8 · p. 16

Abstract

from arXiv · show

Agent skill registries screen what they publish. OpenClaw's security team reported that its scanners overlap on at most 10.4% of combined positives, and 81.9% of flagged skills are caught by one scanner alone. We take that as given, and suggest the open question is not which scanner is right but which is being asked. Each answers a form of "is this skill malicious?", which is what it was built for. "Is this action permitted here, by this operator, right now?" is not one it is designed to express. We report three measurements over 66,192 public ClawHub skill versions. First, 705 skills across 135 distinct publishers that every scanner and the registry's judge rate clean nonetheless instruct an action prohibited by CIS Control 2.7 and NIST SP 800-53 CM-11. A hand audit of 100 puts our detector at 92% precision and found no marker of malicious intent. One publisher contributes 506 of the 705, so we report the distribution with the count. This is reproducible from public artifacts. Second, of 144 commands a live agent executed in a sandbox while following real skill documentation, 34.7% carried a consequence class absent from that document. Third, over 53 cleared skills documenting an action no clean record earns, the agent reached for one in 23 and the gate stopped all 23. The harness and every recorded command are released. We offer one design for that gap: a deterministic resolver with no model in the decision path, feeding a per-(resource, class) trust ledger whose promotion thresholds derive from the operator's stated risk tolerance. Ten clean approvals cannot exclude a true failure rate of 25.9% at 95% confidence. We price the gate's interruptions across a spectrum of operator policies rather than quote one false-positive rate, since friction is a property of the policy, not the gate. We release a 64-case obfuscation benchmark; ours resolves 52%.

1 Introduction

Publish-time screening asks whether a skill is malicious, but the paper argues that action permission depends on the operator and context. It measures gaps between that screening and runtime behavior, then proposes action-level governance.

1.1 A second question about the same artifact

  • Scanner verdicts concern artifact maliciousness, whereas whether an action is permitted depends on the operator’s policy and facts unavailable at publish time.Source: §1.1 · pp. 2–3

1.2 Contributions

  • 705 skills across 135 publishers were rated clean by every scanner and the judge while instructing actions named as prohibited by CIS Control 2.7 and NIST SP 800-53 CM-11.One publisher accounts for 506 of the 705 skills.Source: §1.2 · p. 3
  • 34.7% of 144 commands issued by a live agent carried a consequence class absent from every code block of the skill document.The harness and per-command records are released.Source: §1.2 · p. 3
  • Across 53 cleared skills documenting an action no clean record earns, the agent reached for such an action in 23 cases, and the unmodified gate held or blocked all 23.Source: §1.2 · p. 3
  • The proposed control combines a deterministic resolver with a per-lane trust ledger and autonomy thresholds derived from the operator’s risk tolerance.The resolver is measured fast enough to sit on every tool call.Source: §1.2 · p. 3
  • The released 64-case benchmark covers nine semantics-preserving rewrite techniques; the implementation resolves 52% macro-averaged over techniques and 75% microaveraged.Source: §1.2 · p. 3

2 OATS: Resolving Actions, Not Judging Artifacts

OATS places an action-level permission decision before execution, using a deterministic resolver and a resource–class trust ledger; the section reports what this design covers, costs, and leaves exposed.

2 OATS: Resolving Actions, Not Judging Artifacts

  • OATS asks whether an agent with a given track record may perform a concrete action class on a resource, unattended under the operator’s policy.The resolver does not read the skill’s prose or issue a verdict about the skill.Source: §2 · p. 3

2.1 The deterministic resolver

  • The resolver maps commands to 33 consequence classes, defaults unmatched commands to generic shell_exec, and selects the maximum severity when predicates overlap.It reads only the command string, not documentation, ledger state, time, or a model; identical classifications across different ledger histories support replayability.Source: §2.1 · pp. 3–4

2.2 Threat model, and what ρ does not withstand

  • The resolver assumes benign command recomposition, not an adversary who knows its grammar and writes commands to evade it.The paper limits its non-adversarial measurements to that setting and treats hardening against command-level obfuscation as future work.Source: §2.2 · pp. 5–6
  • 52% macro-average over eight rewriting techniques; micro-averaging yields 75% when the identity control is excluded, because fully resolved techniques contribute more cases.A case counts as resolved when its rewritten command receives a class at least as severe as the base action; the scoring rule and exact match agree on this suite.Source: Table 1 · p. 5
  • Wrapping and chaining resolve completely, but staged fetch-then-execute resolves at 0% because the dangerous relationship spans commands.A per-command classifier cannot detect properties that arise only across commands; addressing this requires shell-grammar resolution and state across calls.Source: §2.2 · pp. 5–6

2.3 Measured cost

  • 67.6 ms median and 81.1 ms p95 end-to-end; resolution is under roughly a millisecond, while receipt storage—not decision latency—scales badly in corpus runs.Earlier isolated-resolution figures were removed because their internal microbenchmark was not released or reproducible.Source: §2.3 · p. 6

2.4 The trust ledger

  • The trust ledger is keyed by resource and class, but the shipped promotion threshold is a single operator-wide default of 10, not the proposed per-class threshold.Thirteen classes never graduate; the paper says the flat default is 3–15× too permissive relative to its derived thresholds.Source: §2.4 · p. 7

3 Data and Method

The study uses the full 66,192-version public dataset and the shipping client to measure policy-relevant instructions. Its counts are bounded by the corpus’s loss of command boundaries and by explicit extraction choices.

3 Data and Method

  • 66,192 skill versions from OpenClaw’s MIT-licensed dataset are classified with the unmodified shipping client.The dataset contains four splits, four scanner verdicts, and redacted SKILL.md content.Source: §3 · p. 7
  • 705 is a plain count of skills, with no deduplication; publisher counts and distributions let readers assess repeated instructions from one vendor.An earlier claim of publisher-level deduplication and counts 2–3× larger is withdrawn.Source: §3 · p. 7
  • The dataset has no newline characters, so counts are at skill granularity and cannot establish whether actions occur “in a single command.”Fenced blocks survive, but command boundaries within them do not; the earlier abstract claim about 657 skills was withdrawn.Source: §3 · p. 7
  • The newline-collapse concern was that unrelated URL and shell text could become adjacent and produce false positives.The analysis tests the syntactic basis of detections to assess that specific concern.Source: §3 · p. 7
  • 98.8% of remote-execution detections rest on constructs that survive newline collapse intact.The check found explicit pipes or process substitution in 87.1%, single-token remote installs in 11.4%, and eight remaining blocks resting on neither.Source: §3 · p. 8
  • Excluding secret_change moves the count from 705 to 706, a factor of 1.00.The class is excluded for continuity with its documented rationale; an earlier claim of roughly 2× inflation was an extraction artifact.Source: §3 · p. 8

4 Finding: Clean Skills Instructing Forbidden Actions

Among skills rated clean by all three scanners and the judge, 705 across 135 publishers instruct actions many operators forbid unattended. The count describes documented instructions, not observed execution, and is concentrated in one publisher.

4 Finding: Clean Skills Instructing Forbidden Actions

  • 705 skills across 135 publishers rated clean by every scanner and the judge nonetheless instruct actions named as prohibited by the cited controls.Table 2 counts policy-relevant capability, not malice; its class-occurrence rows total 709 because four skills appear in two rows.Source: §4 · p. 8, Table 2 · p. 8
  • 1,574 additional skills have incomplete scans, so “nothing objected” there can mean “nothing looked.”The authors report the unanimously clean population rather than treating incomplete scan coverage as equivalent evidence.Source: §4 · p. 8
  • 506 of the 705 skills come from one publisher, oomol; excluding it leaves 199 skills across 134 publishers.The median publisher contributes one skill, and 117 publishers contribute exactly one.Source: §4 · pp. 8–9
  • 152 skills from 93 publishers remain when remote execution is restricted and the dominant publisher is removed.Those skills carry 102 distinct command strings and fetch from 48 distinct domains.Source: §4 · p. 9
  • 92 of 100 randomly sampled detections were correct, giving 92.0% precision and an estimated 598–680 true positives among the 705.The eight errors involved fetches that only format responses, ambiguous protected-branch merges, or ordinary package installation.Source: §4 · p. 9
  • None of the 92 correct detections in the audit showed the specified markers of malicious intent, but the authors do not claim the skills are benign.A single adjudicator reviewed documented code blocks, and the sample was dominated by repetitions from one publisher; clean ratings do not verify benignity, and scanner recall is unpublished.Source: §4 · p. 9
  • The 705 count measures documented instructions, not actions agents actually took.The measurement applies the detector to fenced code blocks; the documentation is its only input in this section.Source: §4 · p. 9
  • 23 of 53 live-agent runs reached for a documented never-graduating action, or 43.4%.Combining that rate with 92% precision yields an estimate of about 281 firings in one agent’s four-turn behavior, distinct from the 705-skill population used to scope a control.Source: §4 · p. 9

5 Finding: The Scanned Artifact Is Not the Executed Artifact

In a narrow sample of skills with specific documented capability envelopes, live-agent commands often carried consequence classes absent from the documentation. The authors distinguish that result from string mismatch and report that the observed gap did not produce higher-severity classes in this sample.

5 Finding: The Scanned Artifact Is Not the Executed Artifact

  • The earlier run, using an unpreserved harness, included 93 commands over 40 skills, with 36.6% class-absent commands; Fisher’s exact test on class-absent counts found no detectable difference between runs (p = 0.78).Source: §5 · p. 10
  • Among skills drawn one per publisher from the unanimously-clean population whose fenced blocks resolved to at least one class more specific than generic shell_exec, executed consequence class was absent from every documented block in 34.7% of cases.These documents were roughly 1% of publishers’ first skills, a deliberately narrow slice.Source: Table 3 · p. 10, §5 · p. 10
  • 50 class-absent commands fell in 14 of 39 skills, and agents recomposed documented steps by chaining checks, adding flags, or splitting commands.One agent turned a documented curl-to-bash step into a staged fetch and inspection after the first attempt failed.Source: §5 · p. 10
  • For 34.7% of executed commands, the consequence class appeared in no documented code block, so reviewers reasoning about the document’s capability envelope would not have anticipated those commands.Source: §5 · p. 11
  • All 50 unanticipated commands resolved to generic shell_exec, with no observed case where recomposition produced a higher-severity class than the document contained.The authors state this bounds the result: the document is a proxy, but it did not fail in a dangerous direction in this sample.Source: §5 · p. 11

6 Experiment: Live Runtime Verification

In a live-agent run, the gate held or blocked every attempted never-graduating action, while the number of interruptions depended on operator policy. The experiment also audits hold accuracy and shows that identical documented steps can resolve to different classes when execution changes.

Method

  • The live-agent study gave each skill one run of at most four turns, with its SKILL.md in the prompt, and executed only commands the gate permitted inside a throwaway container.Every command was recorded in live_agent_runtime.jsonl.Source: §6 · p. 11

Live runtime results

  • 23 of 53 skills with commands prompted an attempt at a never-graduating action, and the gate held or blocked all 23 before execution.The attempts were remote execution in 22 cases, credential access in 2, and a destructive command in 1.Source: §6 · p. 11

Run comparison

  • An earlier version reported 4 of 54 (7.4%); Fisher’s exact test places the two runs apart (p = 1.6 × 10−5), but multiple changes prevent treating their comparison as controlled.Source: §6 · p. 11

Interruption load

  • 170 decisions over 53 skills averaged 3.2 interruptions per skill, including 2.8 holds and 0.4 blocks; 145 decisions (85%) were generic shell_exec.Reviewing every action retains all 3.2 interruptions; allowing reads, docs, tests, and ordinary shell in the repository removes 145 of 170 and leaves 0.5 per skill, while 25 never-graduating-class interruptions remain.Source: §6 · pp. 11–12, Table 4 · p. 12
  • At ϵ = 5%, shell_exec requires 59 consecutive clean approvals and 392 expected approvals given resets, equivalent to about 143 skill installations at this workload’s rate.The paper leaves whether that burden is tolerable to operators; the cost is approvals, not skills, so ordinary work in the same repository can reach the threshold sooner.Source: §6 · p. 12

Hold audit

  • A 50-hold audit found the resolved class correct in 49 cases (98%, 95% CI [89.4%, 99.9%]), including two credential-access holds; a heredoc was falsely classified as destructive.The false positive occurred because dangerous content being written matched a pattern intended to detect a command being run; the paper says this defect is not fixed.Source: §6 · p. 12

Limitations

  • The experiment used one agent, model, and prompt in fresh Debian containers, and the direction of the container’s effect on reach is undetermined.The authors also priced three selected operator policies rather than surveying which policies real operators choose.Source: §6 · p. 12

Runtime resolution

  • A documented install step resolved as ordinary on its first attempt but as remote code execution when retried through a third-party mirror.The resolver classifies what happened in execution, not what the document said would happen.Source: §6 · p. 13

7 What Each Layer Sees

Registry scanners and a runtime action gate address different inputs and questions, so the paper argues that their blind spots are complementary rather than that one replaces another. It describes structural limits, reports their observed overlap, and proposes using registry verdicts as ledger inputs.

7 What Each Layer Sees

  • Table 6 compares five layers by the questions they answer and what falls outside their view, rather than by a shared accuracy score.A reputation service needs history, a document review cannot know which capabilities run, and an action gate sees only resolved actions.Source: Table 6 · p. 13, §7 · pp. 13–14
  • A command-level gate cannot detect a fixed recipient in a document, a harmful behavior assembled from individually ordinary tool calls, or over-broad access visible only across a whole skill.The paper says these are shapes artifact-level capability review is built to catch, and that a runtime gate is not a semantic scanner’s superset.Source: §7 · p. 14

7.1 Why the blast radius is wider together

  • The layers compose rather than compete: artifact-level signals can miss forbidden actions, while a runtime gate cannot see harms that never resolve to a command.The paper gives the example of a benign artifact driving an action an operator forbids, and of the same documented install resolving differently on separate attempts.Source: §7.1 · p. 14
  • The authors withdraw an earlier Jaccard-overlap inference because different base rates drive that statistic and the reported comparison used a mismatched flag population.Source: §7.1 · p. 14
  • Across 66,192 skills, OATS and registry signals overlapped in 680 cases versus 754 expected under independence; ϕ = −0.016 and the reported effect was negligible.The authors say this does not establish independence and that their composition argument instead rests on structurally different inputs.Source: §7.1 · p. 14
  • The proposed composition rule makes a non-clean registry verdict an input to the ledger, allowing an operator policy to keep every lane for that skill held or never-graduating.The paper presents this as a mechanism, not a measured result, because evaluating against the scanner-defined population would be circular.Source: §7.1 · p. 14
  • Two unshipped predicates fired on 4.4% and 1.5% of unanimously clean skills, while the shipped remote-execution predicate fired on 2.25%; the comparison is indicative, not strictly comparable.The candidate predicates were not precision-audited, unlike the shipped predicate, whose firings the authors report as 92% precise.Source: §7.1 · p. 15
  • For skills outside registries, such as those arriving from GitHub, private registries, or shared files, the runtime layer may be the only layer present.Source: §7.1 · p. 15

8 Graduated Autonomy as Operator Policy

This section derives autonomy thresholds from an operator’s tolerated failure rate, then examines their approval costs and the limits of the guarantees they provide.

8 Graduated Autonomy as Operator Policy

  • Zero failures across N independent trials bounds ϵ through (1−ϵ)N = δ, with N set by the tolerated class failure rate ϵk.The rule is the exact inversion of the one-sided Clopper–Pearson bound.Source: §8 · p. 15
  • Ten clean approvals is a chosen rather than derived rule; at δ = 0.05, it leaves a true failure rate of 25.9% unexcluded.That figure is a one-sided 95% upper confidence bound, not the probability that the next action fails.Source: §8 · p. 15
  • At 95% confidence, a flat threshold of 10 demands 3–15× too little evidence.The tolerance bands are a proposal; the implementation applies one threshold to every class.Source: Table 7 · p. 16
  • Ledger trials are correlated and human-approved, so N is a floor and bounds failure under review, not unattended operation.The authors say the ledger alone cannot close the gap between those failure rates.Source: §8 · p. 16
  • At ϵ ≤0.5%, 1,000 unattended actions imply 5 expected failures and a 99.3% chance of at least one; 100 lanes imply a 99.4% chance of at least one wrong graduation.Each lane graduates independently at δ = 0.05; operators can instead spend δ across lanes.Source: §8 · p. 16
  • Graduation requires N consecutive clean approvals; the expected costs are 202 for documentation writes and 965 for business logic, 6–7× N.Several hundred approvals may be amortised over months on high-frequency, low-consequence lanes, but not on rarer ones.Source: §8 · p. 16
  • ϵk encodes operator-stated tolerance, while no finite N licenses classes for which a clean record cannot make an unrecoverable action recoverable.An operator who needs a failure budget per quarter should compute N from that budget, not treat ϵk as a system-level guarantee.Source: §8 · p. 16, Table 7 · p. 16

9 Limitations and Defects in Our Own Instrument

The authors qualify their detector’s evidence, evaluation scope, and runtime measurements, and document defects in the instrument and measurement code that changed reported results.

9 Limitations and Defects in Our Own Instrument

  • A Clean verdict is not verified benign: the registry’s automated verdict lacks adjudicated truth, and scanner recall on this corpus is unpublished.The authors therefore cannot bound how many genuinely dangerous skills received clean verdicts.Source: §9 · p. 17
  • Live evidence is limited by broad intervals, one model on one day, and one frozen registry snapshot; a working rerun changed the count from 4/54 to 23/53.The earlier run failed to reach the install step.Source: §9 · p. 17
  • The authors deliberately report no OATS detection rate against malicious skills because that would evaluate a runtime action gate on a publish-time malware benchmark.They describe the limitation qualitatively instead.Source: §9 · p. 17
  • The hand audit measured 92% precision on 100 sampled skills, but a single rater’s arguable calls lower sensitivity to 89% or 86%.The 89% and 86% figures follow from flipping three or six arguable calls against the detector; the earlier 73.6% figure is superseded.Source: §9 · p. 17
  • The instruction detector has no measured recall: it reads fenced code blocks, so the 705 flagged skills are a floor, not a population estimate.Without a labelled subsample, the authors cannot say whether it finds 5% or 95% of skills that instruct forbidden actions.Source: §9 · p. 17
  • Nine instrument defects were found; the non-shell-fence bug moved 110,319 fences out of the shell population.Other defects included a shell-classification bypass, false-positive regressions, excessive history fetching, and a path misclassified as leaked credentials.Source: §9 · pp. 17–18
  • The measurement script silently omitted two never-graduating classes, and content inside heredocs or comments could trigger a dangerous-pattern match; the latter remains open.Two missing class labels could therefore never appear in Table 2.Source: §9 · p. 18
  • Classifier errors and timeouts were silently discarded, as were 45 blocks truncated at 2,000 characters; the scripts now report these counts.Two of the first six defects surfaced only at corpus scale, while three recent ones surfaced through adversarial review of the measurement code.Source: §9 · p. 18

10 Related Work

The paper locates its runtime gate among reference-monitor principles, application guardrails, registry-security research, and emerging runtime governance, while identifying graduated scrutiny as an open question.

10 Related Work

  • The reference monitor ρ runs on every tool call, uses no model in its decision path, and keys its ledger by resource and class; tamperproofness and adversarial resistance remain incomplete.The authors present the work as an implementation and measurement, not a novel reference-monitor principle.Source: §10 · p. 18
  • Unlike conversational guardrails that place a model or scripted policy between user and model, OATS sits between a chosen action and the system it would change.The paper identifies per-decision model latency as a cost that confines model-consulting rails to conversational turns rather than every tool call.Source: §10 · p. 18
  • ClawHub ranking attacks target pre-install discovery, while trajectory-assurance work addresses sequences of permitted actions; OATS enforces per action, not per trajectory.The authors identify temporal constraints as a possible extension of ρ’s policy language.Source: §10 · pp. 18–19
  • The authors note a commercial runtime-governance tier but say they have not found a published account of when such controls safely relax scrutiny.They qualify that conclusion because relevant logic may exist in unpublished products or accounts they have not seen.Source: §10 · p. 19

11 Conclusion

The conclusion distinguishes publish-time maliciousness screening from permission to act, reports documentation gaps in observed agent behavior, and proposes a runtime control while acknowledging what it cannot govern.

11 Conclusion

  • Across 144 executed commands, 34.7% had a consequence class absent from every code block in the skill documentation.The paper frames maliciousness screening and operator permission as different questions.Source: §11 · p. 19
  • Across 53 cleared skills documenting an action no clean record earns, a live agent reached for such an action in 23.Source: §11 · p. 19
  • The proposed control combines a deterministic resolver measured at 67 ms end to end, a per-(resource, class) ledger, and promotion thresholds derived from operator-stated tolerance.The authors do not claim this is the only or best design.Source: §11 · p. 19
  • A gate cannot resolve harm from hardcoded recipients, undisclosed scope, or persuasive instructions; artifact-level scanning covers that surface instead.Two candidate predicates that might have widened the gate’s reach failed its false-positive bar and were not shipped.Source: §11 · p. 19

Availability

Availability is mixed: corpus results can be re-derived from public artifacts, while live-agent counts are not deterministic. The released resolver permits repeatable class assignments, but proprietary predicates limit auditability.

Availability

  • The corpus study can be re-derived from OpenClaw’s public dataset and the released script, which runs a released implementation rather than a private build.The scanner-agreement figures can also be reproduced with two SQL queries in OpenClaw’s browser console.Source: Availability · p. 19
  • The released resolver implementation contains no model in its decision path, so reruns reproduce the same classes.The implementation used is pheo-oats 0.5.3; the corpus run downloads roughly 1.6 GB and takes several hours.Source: Availability · p. 20
  • The resolver is proprietary, so auditors can re-execute it but cannot inspect its predicates; the published class set and policies do not establish whether a command was classified correctly.The authors distinguish reproducibility by re-execution from auditability.Source: Availability · p. 20
  • The released live-agent scripts and command records require deliberate execution because they fetch and run remote code; without execution, the study measures the harness instead.Execution is off by default.Source: Availability · p. 20
  • Live-agent counts are observations rather than constants: reruns can differ, with divergence figures agreeing across runs (p = 0.78) but runtime reach rates differing (p = 1.6 × 10−5).The authors provide the method, raw records, and a comparison with an earlier run.Source: Availability · p. 20
  • Pheo Inc. evaluates its own product against a bar it set, without a competing implementation, and releases a benchmark to enable outside comparison.The authors identify this as a disclosure.Source: Availability · p. 20
Loading 2609.12001v1…