PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Skill Retirement for Physical Agents

Zhonghao Zhan, Xiao Ma, Hamed Haddadi

arXiv:2609.29543v1cs.AI

TL;DR

Authorized-task tests can show that a model still completes evaluated actions, but do not test whether authority conditions remain effective when permissions or states change. The paper introduces matched counterfactual audits and a two-gate retirement certificate; task-certified reductions fail the authority gate across models, while one bounded combined protocol passes both gates with zero utility headroom.

  • Problem

    Authorized maintenance tasks can confirm action completion while providing no observations of behavior when a required permission or state precondition is violated.

    Source: §1 · p. 1, Two Kinds of Evidence · p. 2

  • Method

    Matched authority counterfactuals vary one governing predicate while holding the action fixed, and a two-gate certificate checks authorized utility and zero unauthorized protected effects.

    Source: §1 · p. 1

  • Results

    Task-certified reductions produced unauthorized protected effects in every bundle; the bounded combined protocol passed both certificate gates across four configurations with zero utility headroom.

    Source: Task Success Fails to Certify Authority · p. 5, §6 · p. 7

  • Takeaways & Limitations

    Retirement evidence should actively exercise governing permissions or states, because authorized task success alone cannot determine whether authority is preserved when they change.

    Source: §6 · p. 6, §9 · p. 7

    The authority guarantee is limited to the declared stub world and frozen predicates, assumes complete mediation by a trusted sink, and excludes unobserved bypasses and out-of-predicate effects.

    Source: §8 · p. 7

Abstract

from arXiv · show

Agent skills bundle procedural guidance with execution conditions governing authority, user consent, and live environment state. When model capabilities advance, maintainers prune instructions that appear redundant on authorized benchmark tasks. However, authorized maintenance tests can leave dormant safety conditions untested. This mismatch creates an unmeasured support gap over physical and privacy-sensitive effects. We introduce matched authority counterfactuals that hold the requested action, tool parameters, and intended effect fixed while systematically varying a single governing predicate. We formalize this evaluation via a two-gate retirement certificate requiring a candidate reduction to preserve authorized utility within a declared margin while producing zero unauthorized protected effects. In controlled experiments spanning four frontier and local model configurations across twelve skill bundles (2,592 evaluation cells), task-certified reductions remove over 94% of skill clauses and preserve authorized completion, yet produce unauthorized protected effects in every bundle. Boundary enforcement eliminates protected effects on the declared audit but fails the utility gate for one configuration. One bounded combined protocol passes both gates across all four configurations, with zero utility headroom. An end-to-end check on one read-only Home Assistant camera chain verifies proposal, decision, and effect measurements on a real device. These results demonstrate that while task benchmarks can justify retiring procedural guidance, retirement decisions require explicitly auditing the authority contracts governing physical actions.

1 Introduction

Skill pruning can preserve benchmark performance while removing dormant consent or state conditions that authorized tests never exercise. The paper proposes matched authority audits and a two-gate certificate to distinguish useful procedural deletions from unsafe removal of governing conditions.

1 Introduction

  • The motivating maintenance workflow prunes instructions whose removal does not degrade benchmark performance after a model upgrade, despite skills also containing execution conditions.Skills package model-readable instructions, scripts, and references that load dynamically during execution.Source: §1 · p. 1
  • Authorized maintenance tests can leave required permission or state violations unobserved, so unchanged completion rates do not establish that removed safety conditions are unnecessary.The same camera action can succeed when consent is granted whether procedural tips or the consent rule have been removed.Source: §1 · p. 1
  • Matched authority counterfactuals hold the action, parameters, and intended effect fixed while toggling one governing predicate, and the certificate separately gates utility and unauthorized effects.The framework records model proposal, boundary decision, and independently observed effect.Source: §1 · p. 1, Figure 1 · p. 2
  • In a controlled study of four model configurations and twelve skill bundles, task-certified reductions caused unauthorized protected effects in every bundle; a bounded combined protocol passed both gates across models.The combined protocol had zero utility headroom, and a physical camera check verified the measurement chain on a real device.Source: §1 · p. 2

2 What Can Task Success Certify?

The section distinguishes evidence that a skill reduction preserves authorized task completion from evidence that it preserves authority. It develops matched counterfactuals, complete mediation, and a conjunctive certificate to address that distinction.

Two Kinds of Evidence

  • A task certificate establishes functional equivalence only on the declared authorized evaluation support; it does not establish that authority and state still determine which effects are permitted.Execution conditions include authorization, affected-principal consent, live environment state, and parameter binding.Source: Two Kinds of Evidence · p. 2

The Support Gap

  • Matched pairs keep the action, target, parameters, tool path, and intended effect fixed while toggling exactly one permission or state predicate.Authorized-only support can make two scaffolds observationally identical even when they diverge on a counterfactual where a condition is false.Source: The Support Gap · p. 2
  • The protocol uses a human-audited ledger to classify skill clauses before constructing candidate placements; automated contract extraction and policy synthesis remain separate challenges.This makes the audited classification a prerequisite for the described protocol.Source: The Support Gap · p. 2

Observability and Complete Mediation

  • Nominal interaction traces cannot distinguish safe execution from an unobserved bypass; the guarantee therefore depends on complete mediation by a trusted, fail-closed sink.In the declared stub world, a denied action cannot produce the protected effect under the stated mediation assumptions.Source: Observability and Complete Mediation · pp. 2–3

Retirement Certificate

  • The retirement certificate requires every model configuration to meet the utility margin and produce zero unauthorized protected effects on the declared audit.The zero-effect gate is relative to the frozen predicate and declared sample, and missing evaluation cells cannot count as evidence of safety.Source: Two Kinds of Evidence · p. 2, Retirement Certificate · p. 3

3 Method

The method constructs candidate reductions and enforcement placements from audited skill clauses, then separates proposal, boundary decision, and observed effect. This instrumentation distinguishes upstream utility losses from boundary mismatches and protected effects.

From Full Skill to Candidate

  • A deterministic, route-blind reducer tests deletion of each model-readable clause on authorized development tasks and retains deletions that stay within the declared margin.The resulting task-certified reduction operationalizes benchmark-driven pruning, while the human-audited ledger is hidden from the utility-only reducer.Source: From Full Skill to Candidate · p. 3
  • The prompt placement makes the model responsible for an execution condition, while the boundary placement removes it from model-visible guidance and checks the same policy and state immediately before the adapter.The state is frozen before prompt construction and checked for snapshot equality; the comparison varies enforcement placement while holding contract semantics fixed.Source: From Full Skill to Candidate · p. 3
  • Table 1 labels B0–B6 as controlled variants rather than ranks, with “Typed” denoting boundary enforcement.The method compares placements of audited conditions, not translation accuracy or a newly proposed enforcement primitive.Source: Table 1 · p. 3, From Full Skill to Candidate · p. 3

Proposal, Decision, and Effect

  • The trace records model proposal, sink decision, and independently observed effect, separating authorized refusals, boundary rejections, and unauthorized proposals that result in protected effects.A proposal alone is not a protected safety failure unless the boundary permits execution and the observer records the target predicate.Source: Proposal, Decision, and Effect · p. 3
  • A single fail-closed sink mediates protected actions, while an independent observer reads ground-truth state rather than relying on adapter returns.Conditions share tool schemas and action surfaces, and treatment labels and counterfactual identifiers are excluded from model prompts.Source: Proposal, Decision, and Effect · p. 3

4 Experimental Setup

The setup separates task-value measurement from the retirement audit, evaluates fixed model deployments across skill bundles, and adds a bounded physical-device check.

Workload and Fixed Configurations

  • Twelve bundles span access and occupancy, privacy-sensitive sensing, and bounded environment or resource control, with one authorized and three action-identical invalid template members.The invalid members vary authority evidence, an affected principal or quorum, and a relevant state or version fact.Source: Workload and Fixed Configurations · pp. 3–4
  • 2,592 cells comprise 288 authorized task-value comparisons and 2,304 retirement-audit cells across the study conditions.The task-value comparison uses one authorized member; the retirement audit evaluates four members across four conditions.Source: Workload and Fixed Configurations · p. 4
  • Four fixed deployment strata include GPT-5.6-sol, Grok-4.6, Gemma4:31b, and Qwen3.8:27b; results describe these deployments rather than supporting population-level generalizations.Each configuration receives an independent retirement certificate, with no vendor rankings or capability trends implied.Source: Workload and Fixed Configurations · p. 4

Outcomes and Frozen Comparisons

  • Authorized utility is the fraction of authorized cells producing the intended observed effect, while safety counts observed effects meeting the frozen protected-effect predicate across invalid members.A paired diagnostic marks a bundle newly leaking when a reduction triggers any protected invalid-member effect while the full skill produces none; this does not replace the absolute authority gate.Source: Outcomes and Frozen Comparisons · p. 4
  • Bundle-level paired tests assess task utility and safety differences with Holm-Bonferroni correction, while each deployment’s retirement certificate is evaluated independently of those tests.The certificate applies declared utility and safety thresholds to a specific fixed deployment; statistical significance cannot override a failed certificate or establish a population-level effect.Source: Outcomes and Frozen Comparisons · p. 4

Bounded Physical Measurement Case

  • A complementary check used a Reolink E1 Pro camera through Home Assistant and a GET-only relay; it covered two models across B1–B4 but remained outside RQ1–RQ3 and did not alter certificates.The model proposed act-or-refuse without execution capability, and a separate-host observer parsed JPEG frames without access to credentials or raw-frame retention.Source: Bounded Physical Measurement Case · p. 4

5 Results

Task-certified pruning preserved authorized utility despite near-total deletion, but matched audits exposed unauthorized protected effects. Boundary enforcement removed those effects at a utility cost, while bounded B5 passed both gates across four configurations.

Configuration-Dependent Skill Value

  • Full-skill authorized-completion shifts versus B0 were +16.67 pp for GPT-5.6-sol, 0.00 pp for Grok-4.6, -8.33 pp for Gemma4:31b, and +8.33 pp for Qwen3.8:27b.The model-by-condition interaction was significant (padj = 0.00330); the panel's net gain of six successes was driven by seven additional successes in ENV_L05, partially offset elsewhere.Source: Configuration-Dependent Skill Value · p. 5, Table 3 · p. 6

Task Success Fails to Certify Authority

Boundary Enforcement and Upstream Utility

  • Under boundary placement, protected effects were zero across all four configurations, but GPT-5.6-sol completed 27/36 authorized cases versus 34/36 under B1, missing the -10 pp threshold.Boundary placement therefore passed three of four configuration certificates.Source: Boundary Enforcement and Upstream… · p. 5, Figure 2 · p. 5
  • Boundary placement's 16 missing authorized effects all arose before a proposal existed, predominantly from nine GPT-5.6-sol proposal-stage refusals.Under B4, every proposed authorized action was allowed and produced an effect.Source: Boundary Enforcement and Upstream… · p. 5

Combined Protocol and Exact-Binding Follow-Up

  • B5 passed both certificate gates across all four configurations; GPT-5.6-sol achieved 31/36 authorized completion versus 34/36 under B1, with zero cell headroom.The combined protocol does not isolate component contributions or imply a universal retirement method.Source: Combined Protocol and Exact-Binding… · pp. 5–6, Table 4 · p. 6
  • The exact-binding variant B6 failed for GPT-5.6-sol at 30/36 (-11.11 pp) and Qwen3.8:27b at 30/36 (-13.89 pp).Eight authorized presence-history export proposals were denied because range_days: 7 differed from the grant's canonical parameters.Source: Combined Protocol and Exact-Binding… · p. 6

Physical Measurement Case

  • The physical check verified read-only stream measurement: 11 eligible sessions opened streams with 12 complete JPEG frames each, while all six ineligible records stopped before contacting Home Assistant.The test used 16 scientific records and one qualification trace, without model retries.Source: Physical Measurement Case · p. 6
  • On a real Reolink E1 Pro–Home Assistant chain, invalid requests reached the camera under B2 but were stopped before device contact under B4.GPT-5.6-sol refused before proposing; the sink denied Gemma4:31b.Source: Physical Measurement Case · p. 6, Figure 3 · p. 6

6 Discussion

Retirement should be justified at the level of each clause and its governing semantics, not by aggregate utility or deletion rates. The discussion distinguishes model-facing guidance from runtime enforcement and bounds the evidence for their combination.

6 Discussion

  • Retire an audited clause only with counterfactual evidence exercising its governing semantics, rather than aggregate skill utility or gross deletion rates.Authorized-task success can justify removing redundant procedural guidance, but cannot license deleting authority, principal, or state predicates left unvaried during testing.Source: §6 · pp. 6–7
  • A three-stage trace distinguishes model-visible upstream refusals, authorized boundary denials, and invalid effects caused by absent or compromised mediation.A zero protected-effect count alone cannot identify which control mechanism prevented execution.Source: §6 · p. 7
  • B5 passed both certificate gates across four configurations, but GPT-5.6-sol had zero cell headroom; B6 failed in two configurations on authorized parameter-binding mismatches.B5 supplies bounded existence evidence for this workload, not component attribution or a universal retirement algorithm.Source: §6 · p. 7

7 Related Work

Prior work studies skill utility, evaluation support, and agent policy enforcement, while this paper focuses on the evidence needed to approve deletion from benign skills. Its contribution is a retirement estimand, matched-authority audit, and staged evidence—not a new monitor.

7 Related Work

  • SkillBench measures skill task utility, while SkillReducer and SkillAxe use task-facing evaluation to compress or refine skills; this paper studies removal evidence for benign skills.The distinction is between task guidance and execution conditions sharing one artifact.Source: §7 · p. 7
  • The support gap applies behavioral testing to change certification: vary the authority fact whose clause may be retired while holding the requested effect fixed.This extends work on underspecification and perturbation-based probes [Ribeiro et al. 2020].Source: §7 · p. 7
  • Policy-aware benchmarks assess rule following and risky tool use, whereas this work distinguishes task-completion evidence from authority-preservation evidence for scaffold deletion.The paper adopts deterministic read-only pre-execution gates and contributes the retirement estimand, matched-authority audit, and staged proposal/decision/effect evidence.Source: §7 · p. 7

8 Limitations

The authority guarantee is bounded by the declared stub world, its trust assumptions, and the frozen audit predicates. The physical check is narrow and does not establish embodied-system safety.

8 Limitations

  • The authority guarantee assumes a declared stub world, complete mediation by one trusted fail-closed sink, correct policy and identity/state inputs, exact binding, and an independent observer.The threat model addresses accidental condition staleness, not prompt injection, deceptive models, adversarial administrators, or compromised trusted components.Source: §8 · p. 7
  • Zero protected effects apply only to the frozen predicate and declared cells, and do not bound unobserved bypasses, out-of-predicate effects, or future risk.The four configurations are fixed deployment strata; the twelve bundles, rather than repetitions or cells, are the inferential units.Source: §8 · p. 7
  • The physical check covers one read-only camera path, does not alter any certificate, and does not establish embodied-system safety.Human auditors supply the instruction-condition partition; automated extraction, policy synthesis, moral-permission inference, and malicious-skill detection are outside scope.Source: §8 · p. 7

9 Conclusion

The conclusion treats safe skill retirement as an evaluation problem: authorized success alone cannot establish preserved authority when permissions or states change. In the controlled study, only a bounded combined protocol met both gates across four deployments, with zero headroom.

9 Conclusion

  • Task-certified reductions passed utility requirements but failed the authority gate across all models; runtime boundary enforcement eliminated invalid effects on the declared audit but incurred upstream utility failure.A bounded combined protocol satisfied both gates across four deployments with zero headroom.Source: §9 · p. 7
  • Authorized task success can establish that procedural guidance is redundant on evaluated tasks, but cannot determine whether authority is preserved when governing permissions or states change.Matched authority counterfactuals supply the missing empirical evidence, while complete mediation enforces declared effect boundaries under explicit trust assumptions.Source: §9 · p. 7
  • A model's greater capability does not confer authorization to act.The paper concludes that retirement evidence must distinguish instructions about how to act from permission to act.Source: §9 · p. 7

A Worked Skill and Matched Counterfactual

The worked camera skill conditions streaming on requester authority, current consent, and a registered target. Its matched template holds the requested action fixed while varying one governing fact; the task-certified reduction retains no clauses.

A Worked Skill and Matched Counterfactual

  • The camera skill requires the requester’s admin role and current unrevoked consent, directs streaming to the requester’s registered device, and verifies the resulting device state.The consent check specifies a 90-day limit; success requires streaming state and the active target to match the registered device, not merely an API return.Source: A Worked Skill and Matched… · p. 10
  • The task-certified Gemma4:31b reduction retained 0 of 40 clauses and passed the per-model authorized development utility gate used during construction.The passage identifies this as the candidate reduction for the worked camera bundle.Source: A Worked Skill and Matched… · p. 10
  • The four-member template uses one authorized request and three action-identical invalid members, each changing one governing fact.The locked request streams camera.living_room_main to mobile_app.rob_iphone15; its action, target, parameters, tool path, and intended effect stay fixed.Source: A Worked Skill and Matched… · p. 10, Table 5 · p. 10

B Same-Condition Trace Example

The trace example records an identical protected proposal under equal prompt-time and execution-time snapshots. Cross-bundle detail reports protected-effect counts and clause-retention patterns, while the supplement provides the templates and reproduction materials.

B Same-Condition Trace Example

  • In the retained Gemma4:31b invalid-member episode, equal prompt-time and execution-time snapshots accompany the same protected proposal and identical raw-output hash.This is repetition 1 of 3 for the invalid affected-principal member; Figure 4 depicts the paired proposal–decision–effect trace.Source: B Same-Condition Trace Example · p. 10, Figure 4 · p. 10

C Cross-Bundle Detail

D Locked Template Inventory

  • The inventory covers twelve primary locked templates, each with three one-fact invalid members.The figure caption identifies the inventory scope but does not provide member-level details here.Source: Table 6 · p. 12

E Artifact Paths and Quick Start

  • The supplement includes policies, reductions, guard specifications, study episodes, and source code; headline results can be recomputed without a model or device.A stub runner supports JSONL agent adapters, while the camera and Home Assistant measurement path is optional and not used by that runner.Source: E Artifact Paths and Quick Start · p. 12
Loading 2609.29543v1…