PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillMarker: A KGW-Based Watermarking Method for Agent Skill Ownership Verification

Alan Tang, Lifeng Wang, Gang Wu, Tao Zhang, Yantao Jia, Xiaofeng Zhu

arXiv:2610.15725v2cs.CRcs.AI

TL;DR

Plaintext skills expose reusable expertise to copying and leakage, while their precise instructions make watermark embedding difficult. SkillMarker combines quality-constrained rewriting with signal-oriented candidate search; across four datasets, it reports negligible average task-performance degradation alongside detection of short copied passages and merged skills.

  • Problem

    Plaintext distribution exposes reusable agent skills to copying and leakage, while their compact, accuracy-sensitive instructions constrain watermark rewriting.

    Source: §1 · pp. 1–2

  • Method

    SkillMarker combines quality-constrained rewriting that preserves operational requirements with candidate search for stronger watermark signals among acceptable rewrites.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Across four datasets, watermarked skills have negligible average task-performance degradation; window detection finds copied passages as short as 1,536 consecutive tokens, with 72.2–77.8% detection after merging with background skills.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Keyed watermarks support skill ownership verification and leakage tracing, including when instructions are copied in whole or incorporated into other skills.

    Source: §1 · p. 2

    Nominal binomial probabilities require independent calibration before they can be interpreted as empirical false-positive guarantees.

    Source: §2.4 · p. 4

Abstract

from arXiv · show

Agent skills encode operational procedures and domain expertise as reusable knowledge assets, yet their distribution in plaintext exposes them to unauthorized copying and leakage. However, practical mechanisms for skill ownership verification and leakage tracing remain underdeveloped. Watermark embedding before distribution offers a potential solution, but structured, compact skill instructions impose stringent requirements for factual and procedural accuracy. To address these challenges, we introduce SkillMarker, a KGW-based watermarking method that jointly targets functionality preservation and strong watermark embedding. SkillMarker unifies quality-constrained rewriting with signal-oriented candidate search. The former separates essential knowledge from flexible expression to preserve instruction fidelity; the latter explores alternative expressions to obtain a stronger watermark signal under preservation constraints. We release an open-source toolkit that integrates watermark embedding, quality auditing, and detection to support the protection of agent skills. SkillMarker targets ownership verification and leakage tracing in distributed skill ecosystems, where content may be copied in whole or in part, or incorporated into other skills. Across four datasets, SkillMarker negligibly affects downstream task performance on average. In our copy-paste tests, window-based detection consistently identifies copied passages containing as few as 1,536 consecutive tokens. Even after merging with one or three background skills, SkillMarker retains watermark detection rates of 72.2--77.8\%.

1. Introduction

Plaintext distribution makes reusable agent skills vulnerable to copying and leakage, while compact, accuracy-sensitive instructions leave little room for watermark rewriting. SkillMarker combines constrained rewriting and candidate search to preserve functionality while supporting ownership verification and leakage tracing.

1. Introduction

  • Plaintext skills make reusable operational knowledge easy to copy, redistribute, or incorporate into other skills.Skills package domain expertise into reusable instructions and resources that agents load on demand.Source: §1 · p. 1
  • Watermarking compact, structured instructions must preserve precise requirements while finding enough rewriting freedom to embed a strong signal.Small changes to references, decision conditions, or required actions can change agent behavior; weak logit bias may yield little signal, while stronger bias can alter essential instructions.Source: §1 · pp. 1–2
  • SkillMarker uses quality-constrained rewriting to protect operational requirements and signal-oriented search to select stronger-signal candidates that pass preservation checks.It adapts KGW for skill ownership verification.Source: §1 · p. 2
  • A keyed watermark supports ownership checks and source-version tracing, including when instructions are copied or incorporated into other skills.The toolkit combines embedding, quality auditing, and detection.Source: §1 · p. 2
  • Across four datasets, watermarked skills show negligible average task-performance degradation; window detection finds copied passages as short as 1,536 consecutive tokens, and merged-skill detection rates are 72.2–77.8%.The merge result applies after combination with multiple background skills.Source: §1 · p. 2

2. Method

SkillMarker embeds keyed watermark signals by rewriting editable skill text while preserving protected content and operational requirements. Its candidate search favors stronger signals among acceptable rewrites, and its detector scans suspect text using full-text or window-based tests.

2.1. Problem Setup and Overview

  • The setup associates a watermarked skill with a key and supports later testing of suspect text, including partial skill content.Embedding uses a language model with output-logit access; detection needs the tokenizer and watermark configuration but no model inference.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2. Quality-Constrained Rewriting

  • Code, delimiters, recognized literals, unresolved structures, and short spans are protected, while longer prose spans are rewritten and reassembled in order.Paired examples and explicit constraints preserve facts, conditions, negations, values, identifiers, and layout; independent auditing checks language, semantics, completeness, and usability.Source: §2.2 · pp. 3–4

2.3. Signal-Oriented Candidate Search

  • Beam search explores KGW-biased continuations, then selects preservation-passing candidates by higher green-token rate and lower PPL as a tie-breaker.Candidates are tokenized in reconstructed context; if none pass, the original span is retained, and local selection does not guarantee the highest whole-skill rate.Source: §2.3 · p. 4

2.4. Watermark Detection

  • Full-text and window-based detection score eligible tokens in suspect text; nominal binomial probabilities require independent calibration before serving as empirical false-positive guarantees.Figure 3 compares task performance, token usage, and task time for no-skill, original-skill, and watermarked-skill settings.Source: §2.4 · p. 4, Figure 3 · p. 5

3. Experiments

Across four datasets, watermarking largely preserves downstream task performance; robustness tests assess copied and merged skills, while ablations expose a quality–strength trade-off.

3.1. Experimental Setup

  • The evaluation used four datasets spanning everyday/general and professional scenarios, with no-skill, original-skill, and watermarked-skill settings.Qwen3-8B embedded watermarks, and DeepSeek-V4-Flash executed tasks through Open Code.Source: §3.1 · p. 5

3.2. Downstream Performance and Efficiency

  • Watermarked skills changed task performance by −2.01% to +0.93% relative to original skills across four datasets and retained their gains over the no-skill baseline.On agent-skills, pass rates were 97.46% before and 98.25% after watermarking; on skill-use-bench, they were 59.85% and 60.41%.Source: §3.2 · pp. 5–6

3.3. Watermark Robustness

  • Robustness evaluation covers watermarked passages copied into unwatermarked text and skills merged with background skills, measuring detection in both scenarios and localization for copy-paste.Copy-paste conditions varied copied length, background length, and fragment count.Source: §3.3 · p. 6, §3.3.1 · p. 6

3.3.1. Copy-Paste

  • 1,536-token contiguous copies were all detected by window-based detection in both background lengths; in the 4,096-token background, full-text detection reached only 55.6%.The copy-paste setup used 1,024- or 4,096-token unwatermarked backgrounds.Source: §3.3.1 · p. 6, Figure 4 · p. 7
  • 98.0–99.1% localization recall and 65.4–69.6% precision were achieved for contiguous 1,536-token copies; eight fragments in the longer background reduced detection to 77.8%.Detection windows recovered nearly all copied content while including some surrounding text.Source: §3.3.1 · p. 6, Figure 5 · p. 7

3.3.2. Skill Merging

  • Multi-scale union detection rates were 91.7%, 72.2%, and 77.8% with one, two, and four input skills, versus 58.3%, 47.2%, and 44.4% for single-window detection.False-positive rates for both methods were 0.0%, 11.1%, and 16.7% at those input-skill counts.Source: §3.3.2 · pp. 6–7, Figure 6 · p. 8

3.4. Ablation Study

  • SkillMarker achieved quality 70.97 and mean z-score 3.01; removing fidelity constraints raised z-score to 3.37 but lowered quality to 67.09.Removing beam search, signal-oriented selection, or logit bias reduced mean z-score to 0.65–0.83 despite higher quality scores.Source: Figure 7 · p. 8, §3.4 · p. 8

3.5. Structured Skill Case Study

  • Protected spans in the observability-and-instrumentation excerpts remained exactly unchanged, and the saved full text had nominal p = 4.62 × 10^-9.Long protected blocks show only their opening lines in the excerpts.Source: §3.5 · p. 8

4. Related Work

Prior work covers generation-time and post-hoc watermarking, provenance auditing, and text steganography. SkillMarker addresses the distinct joint requirement of preserving skill instructions while supporting attribution after reuse.

4.1. Watermarking

  • Generation-time watermarking typically attributes model outputs, whereas SkillMarker lets skill developers watermark reusable instructions before distribution, independently of the models that execute them.Prior approaches also watermark released artifacts or traces, but semantic preservation alone does not establish instruction fidelity in skills.Source: §4.1 · pp. 8–9

4.2. Text Steganography

  • Linguistic steganography targets concealed-message recovery, while SkillMarker embeds watermarks directly in distributed skills to support ownership verification and leakage tracing under preservation constraints.The cited steganalysis studies report detection of steganographic text across tested methods and domains.Source: §4.2 · p. 9

5. Conclusion

SkillMarker protects distributed agent skills through quality-constrained rewriting and signal-oriented candidate search. Across four datasets, watermarked skills largely preserve downstream performance, while detection remains affected by copied-text dilution, fragmentation, and background content.

5. Conclusion

  • Across four datasets, watermarked skills largely preserve downstream performance, with relative changes of −2.01% to +0.93% and dataset-dependent token overhead.The method uses quality-constrained rewriting and signal-oriented candidate search.Source: §5 · p. 9
  • Window-based detection improves detection of copied content diluted by unwatermarked text, while fragmentation weakens detection and localization.Watermarks remain detectable after merging with a limited number of background skills, although added background content weakens detection overall.Source: §5 · p. 9
  • Elevated control false positives highlight the need for calibration when applying SkillMarker to ownership verification and leakage tracing.The findings support watermarking as a practical direction for these applications.Source: §5 · p. 9

A.1. Absolute Downstream Results

Table 1 reports performance, mean tokens, and median task time across four datasets and three skill settings; its absolute values underlie Figure 3.

A.1. Absolute Downstream Results

  • Table 1 reports performance, mean tokens, and median task time across four datasets and three skill settings.The table reports the absolute values underlying Figure 3 for the same settings and datasets.Source: Table 1 · p. 12, §A.1 · p. 12

A.2. Main Experimental Hyperparameters

The appendix lists hyperparameters for downstream evaluation and robustness experiments, which use the frozen English watermarked release. It also notes limits in the copy-paste probability treatment and merge-detector parameter selection.

A.2. Main Experimental Hyperparameters

  • Robustness experiments use the frozen English watermarked release, with its embedding settings listed separately from downstream evaluation.The hyperparameter table is organized by experiment.Source: §A.2 · p. 12
  • Downstream evaluation uses DeepSeek-V4-Flash [1m] and the same model for judging, with one judgment per case and 10,000 bootstrap replicates.The listed seed is 20261010.Source: §A.2 · p. 12
  • Robustness watermark embedding uses Qwen3-8B with beam width 5, green fraction 0.75, and logit bias 8.The listed hash key and embedding seed are 15485863 and 123.Source: §A.2 · p. 12
  • Copy-paste tests vary copied lengths from 256 to 2,048 tokens and unwatermarked text lengths of 1,024 or 4,096 tokens.They use 1, 2, 4, or 8 fragments and 1,024-token detection windows with stride 256.Source: §A.2 · p. 12
  • Skill-merging tests use one, two, or four input skills—one target plus zero, one, or three backgrounds—and a multi-scale union of 13 component decisions.The listed single-window length and stride are 3952 and 1027; the probability threshold is p≤0.03332253.Source: §A.2 · p. 12
  • Copy-paste window tests use nominal probabilities without multiple-testing correction, and merge-detector parameters were selected on the evaluation cohort without a held-out test set.These choices constrain the reported probability treatment and the independence of merge-detector evaluation.Source: §A.2 · p. 12

B. Experimental Prompts

The appendix documents the watermark-embedding prompt and skill-merging prompt. The embedding demonstrations illustrate preservation rules, while the merging prompt instructs the system to combine the supplied skills.

B.1. Watermark Embedding

  • The embedding prompt replaces its final user message with the current editable passage; role labels and the bracketed input field are presentation annotations.The prompt text is reproduced from the saved generation configuration.Source: §B.1 · p. 13
  • The embedding prompt asks the rewrite to preserve meaning, key facts, conditions, negations, numbers, technical identifiers, and formatting while allowing other wording to change.The prompt says to output only the rewritten passage in its original language.Source: §B.1 · p. 13
  • The first demonstration automatically writes patterns to skill references only when frequency is at least 3, fixes are consistent, and certainty is 100%; otherwise it keeps them as a staged patch.The user and assistant examples express the same conditions.Source: §B.1 · p. 13
  • The second demonstration treats purely visual ALIGN differences in CHECK-8 and CHECK-11 as reportable but non-blocking, while CRASH/URL issues in CHECK-8 block the gate and enter the §5 loop.The assistant example preserves the distinction made in the user example.Source: §B.1 · p. 13

B.2. Skill-Merging Instruction

  • For merging evaluation, the system message instructs the model to merge the provided skills into one skill.The user message contains the complete input skills.Source: §B.2 · p. 13
  • The merging prompt supplies the complete input skills in their recorded order.The prompt uses a bracketed placeholder for those skills.Source: §B.2 · p. 13

C. Extended Structured Skill Case Study

Figure 9 presents extended excerpts showing preserved protected content alongside rewritten prose that carries a watermark. The examples document literal preservation and detection, not semantic or functional equivalence.

C. Extended Structured Skill Case Study

D. Ablation Settings and Additional Results

The ablations show a balance between watermark strength and holistic quality: SkillMarker does not dominate every alternative. The section also specifies the evaluation setup and the component variants used for comparison.

D. Ablation Settings and Additional Results

  • The ablation evaluates 32 selected skills from four datasets with Qwen3-8B, three fixed seeds per skill and configuration, and equal weighting across skills.SkillMarker uses five-beam search, γ=0.75, and logit bias δ=8.Source: D. Ablation Settings and Additional… · pp. 14–15
  • The variants remove prompt-level preservation constraints, use a single search path, change candidate ranking, score generated tokens, or set logit bias to zero while retaining candidate selection.Signal-oriented selection and reconstructed-text scoring variants replay SkillMarker’s candidate pools; all six configurations retain mandatory literal protection.Source: D. Ablation Settings and Additional… · p. 15
  • SkillMarker achieves quality 70.97 and mean strength 3.01; removing fidelity constraints raises strength to 3.37 but lowers quality to 67.09.The results characterize SkillMarker as offering a balance rather than dominating every alternative.Source: D. Ablation Settings and Additional… · p. 15
  • Removing beam search, signal-oriented selection, or logit bias reduces strength to 0.65, 0.69, or 0.83, respectively, while quality rises to 76.06, 75.50, or 73.06.These component variants therefore contribute watermark evidence at a cost in holistic quality score.Source: D. Ablation Settings and Additional… · p. 15
  • Generated-token scoring yields quality 70.00 and strength 2.98, close to SkillMarker; these means alone do not establish a substantial benefit from reconstructed-text scoring.Source: D. Ablation Settings and Additional… · p. 15

Related papers

Loading 2610.15725v2…