PaperlayerView exact revision on arXiv

Source-linked AI summary

RedAct: Redacting Agent Capability Traces for Procedural Skill Protection

Shuwen Xu, Zhitao He, Yi R. Fung

arXiv:2606.10813v3cs.CRcs.CL

TL;DR

Released agent traces can expose reusable proprietary procedures, creating tension between auditability and skill protection. REDACT combines selective trace rewriting with behavioral watermarking, reducing downstream skill reuse while preserving verifier-critical evidence.

  • Problem

    The paper asks how useful agent traces can be released without exposing reusable protected skills while preserving evidence of downstream reuse.

  • Method

    REDACT combines trace rewriting that abstracts protected procedures with behavioral watermarking for provenance tracking, evaluated on 75 tasks and 154 curated skills.

  • Results

    Across reuse settings, REDACT reduces downstream skill reuse while preserving verifier-critical evidence; key-info guidance drives NST below the No Skills baseline for all three reuse methods.

  • Takeaways & Limitations

    Selective trace protection can reduce procedural capability leakage while retaining evidence needed for auditing and providing empirical provenance signals.

  • Takeaways & Limitations

    The evaluation uses controlled long-horizon tasks with automatic verifiers, so real deployments may involve noisier traces, changing environments, and weaker verifier feedback.

Abstract

from arXiv · show

Users rely on execution traces to observe agent behavior, diagnose failures, and ensure accountability. These traces contain rich procedural detail, including tool invocations, intermediate decisions, and error-recovery logic. Yet this detail can expose private procedural skills, allowing downstream methods to recover key formulas, thresholds, and strategies without access to model weights or skill files. To quantify this risk and evaluate protection, we construct CapTraceBench, a benchmark of 75 specialized long-horizon tasks and 154 curated skills across seven domains. We also introduce RedAct, a protected trace release framework that localizes protected key information, rewrites traces while preserving verifier-critical evidence, and embeds behavioral watermarks for downstream provenance analysis. Across representative trace reuse methods, RedAct reduces normalized skill transfer (NST) from 44.7-67.1% on raw traces to below the no-skill baseline, while preserving audit evidence. Its standalone behavioral watermarks reach 93.6-100.0% true detection with a false alarm rate of at most 1.9%. These results frame public agent traces as security interfaces and show that selective redaction can reduce procedural capability leakage without removing audit evidence.

1 Introduction

Agent execution traces support transparency, debugging, and trust but can expose proprietary procedural skills that downstream parties recover and reuse. RedAct frames this tension as black-box trace disclosure and proposes protected trace release alongside CapTraceBench for evaluating procedural skill protection.

  • Motivation: Proprietary agent skills encode reusable workflows, code templates, tool-use routines, validation scripts, and domain heuristics.These skills support agents built around proprietary data, domain models, and expert-refined workflows.
  • Motivation: Released traces can reveal formulas, thresholds, validation protocols, recovery strategies, and other protected procedures underlying agent effectiveness.This creates a tension between trace auditability and procedural skill protection.
  • Problem formulation: Black-box trace disclosure evaluates whether downstream methods can recover reusable protected procedures using released artifacts without private skill files, model weights, or hidden states.The risk concerns reusable procedural recovery across repeated deployments rather than answer copying.
  • Motivation: Released traces expose private files, curated skills, tool selections, and recovery steps, whereas answer-only releases and generic summaries remove or obscure audit context.This motivates trace-level redaction and provenance analysis.
  • Approach: RedAct combines selective trace rewriting with behavioral watermarking to protect procedural skills while retaining useful execution evidence.The framework assumes skill owners access protected packages while downstream reuse methods observe only released traces.
  • Benchmark: CapTraceBench comprises 75 long-horizon tasks across seven domains and 154 curated skills for evaluating procedural skill protection.The benchmark is presented as the first benchmark for this evaluation problem.

2 Problem Setting

The problem setting models procedural skills as protected knowledge embedded in agent trajectories and compares disclosure policies that expose raw execution, only answers, or audited traces with reusable content abstracted. Downstream reuse methods operate solely on released artifacts, and a successful defense should reduce same-task utility toward the no-skill baseline.

  • Skills and trajectories: A procedural skill s = (m, R) combines an instruction document with auxiliary resources, while Iprot(s) contains reusable formulas, thresholds, tool choices, workflow dependencies, and validation routines.The agent acts on task x with history h and produces a trajectory containing reasoning, actions, observations, and a final output.
  • Disclosure policies: Disclosure policy ϕ maps a trajectory to a public artifact, with raw disclosure exposing most execution, answer-only disclosure revealing y, and REDACT retaining audit evidence while abstracting reusable protected content.REDACT produces the protected trace zRA = ϕRA(x, τ).
  • Disclosure policies: REDACT localizes key procedural knowledge, rewrites trajectories, and injects behavioral watermarks for provenance detection.These operations define the protected-trace release framework described for CAPTRACEBENCH and REDACT.
  • Downstream reuse: Downstream methods access only publicly released artifacts, not model weights, private skill files, hidden states, server prompts, or undisclosed trajectories.The considered reuse methods are single-agent synthesis, multi-agent evolution, retrieval reuse, and trajectory fine-tuning.
  • Downstream reuse: A successful defense should reduce the same-task utility of student policies induced from ϕRA toward the no-skill baseline.The evaluation metrics are defined in Section 5.

3 CAPTRACEBENCH

CAPTRACEBENCH evaluates procedural skill protection under public trace disclosure using 75 specialized long-horizon tasks and 154 curated skills across seven domains. Its protocol tests whether released traces enable downstream reconstruction and reuse of protected procedural knowledge while preserving useful execution evidence.

  • Benchmark scope: CAPTRACEBENCH comprises 75 specialized long-horizon tasks and 154 curated skills across seven domains.The benchmark is introduced to evaluate procedural skill protection under public trace disclosure.
  • Task design: Tasks require multi-step tool use and include human-authored instructions, executable environments, automatic verifiers, and 1–3 specific skills.Examples include DNA sequence analysis, code review, control-system calibration, and scientific computation.
  • Skill construction: Skills are selected for domain specialization, multi-step grounding, and executability through sandboxed code or tool-output verification.The stated criteria target formulas, tools, workflows, or heuristics that task instructions alone cannot recover.
  • Evaluation protocol: Evaluation uses units (s, x, V), where a teacher trajectory and final output are disclosed as a public artifact for constructing a downstream student.Protection is measured by the induced student’s verifier performance in the same task environment, testing reconstruction and reuse of protected procedural knowledge.

4 Methodology

REDACT combines key-information-guided trace rewriting with behavioral watermarking for provenance analysis. Its rewriting preserves verifier-critical audit evidence while abstracting protected procedural details, and its watermarks insert detectable, functionally neutral behaviors into selected traces.

  • Framework: REDACT has two components: key-information-guided rewriting for trace protection and behavioral watermarking for provenance analysis.Figure 2 summarizes the framework.
  • Trace rewriting: The rewriting layer identifies protected formulas, constants, thresholds, tool choices, and validation routines, then human review deduplicates the resulting key-item set before rewriting.The Key-Item Locator operates on the task instruction and skill package, while Human Review merges duplicate items such as equivalent rise-time thresholds.
  • Trace rewriting: The Rewriter abstracts intermediate turns exposing protected items while preserving final outputs, tool-use evidence, and execution order required for auditing.The released trace is positioned between answer-only release and raw trajectories in informativeness and procedural reusability.
  • Behavioral watermarking: Behavioral watermarks use functionally neutral hook families that insert detectable action-observation patterns and activation phrases into eligible protected traces.Injected patterns are constrained to preserve verifier outcomes and the original final answer.
  • Behavioral watermarking: Standalone hooks use fixed trace positions, whereas contextual hooks follow eligible tool results or error observations and therefore depend on execution context.At detection time, REDACT queries a suspected student under the activation phrase and measures true detection rate on matching-watermark students and false alarm rate on the base model.

5 Experiments

Experiments on CAPTRACEBENCH show that raw execution traces transfer substantial procedural skill, especially on hard tasks, while REDACT suppresses this reuse and preserves audit evidence. Behavioral watermarks provide provenance signals, and key-info-guided rewriting outperforms generic rewriting.

  • Benchmark: CAPTRACEBENCH comprises 75 long-horizon tasks across 23 task families and seven domains, with 154 curated skills and automatic verifiers.
  • Raw Trace Reuse: 73.5% average SSR from extracted skills, 73.7% from evolved skills, and 71.8% from retrieval reuse exceed the 68.0% No Skills baseline on raw traces.Oracle Skills reaches 76.5% average SSR, compared with 68.0% under No Skills.
  • Protection Effect: 67.5%, 66.3%, and 65.5% protected-trace SSR for extraction, evolution, and retrieval reuse are no higher than the 68.0% No Skills baseline.NST falls from 44.7–67.1% under raw traces to at most −5.9% after protection.
  • Difficulty Analysis: 6.5–10.5% raw-trace SSR gains on hard tasks exceed the 2.1–2.7% gains on easy tasks, where general competence is more sufficient.Hard-task synthesized skills recover much of the Oracle Skills gain, exposing implementation details, debugging choices, and task-specific routines.
  • Behavioral Watermarks: 93.6% true detection from the Env Check standalone hook demonstrates a clear provenance signal from behavioral watermarks.Detection uses Qwen3-8B and Qwen3-4B students fine-tuned on watermarked trajectories and evaluated on 20 Easy-level tasks with 25 runs per task.
  • Release Integrity: 91.0–96.6% audit-evidence retention and 70.6% key-item removal show that REDACT preserves trace observability while removing protected content.Tool-call count and trace length remain within the 90–110% near-parity band, with 29.4% residual key-item content.
  • Rewriting Ablation: Key-info-guided rewriting drives NST below the No Skills baseline for all three reuse channels, whereas generic rewriting leaves positive NST for extracted and evolved skills.The ablation compares both rewriting strategies using Claude Code with Claude Opus 4.6 and five runs per setting.

6 Related Work

Related work spans tool-using agents and reusable skills, trace distillation and protected disclosure, and watermarking for provenance. RedAct studies disclosure risk from released trajectories and extends behavior-based agent watermarking.

  • Tool-Using Agents and Skill Reuse: Tool-using agents have progressed from reasoning-interleaved action to deployed software workflows, while agent experience can be distilled into reusable skills and workflows.This work studies the resulting disclosure risk from released trajectories exposing reusable procedural knowledge.
  • Trace Distillation and Protected Disclosure: Agent research connects RedAct to model and training-data extraction, while recent studies show proprietary skills and reasoning traces are vulnerable to distillation.Existing defenses primarily protect reasoning text.
  • Watermarking and Provenance: Text and API watermarking detect generated outputs through token-level signals or output fingerprinting, whereas recent agent methods shift provenance detection to behavior.RedAct builds on behavior-based agent watermarking.

7 Conclusion

The paper frames public agent traces as security interfaces that enable transparency and auditing while potentially exposing reusable procedural skills. It introduces CAPTRACEBENCH and REDACT to evaluate and mitigate this risk through selective rewriting and behavioral watermarking while preserving verifier-critical evidence.

  • Security framing: Public agent traces support transparency and auditing but can expose reusable procedural skills.The paper formalizes this exposure as black-box trace disclosure.
  • Protected release framework: REDACT combines selective rewriting with behavioral watermarking to reduce downstream skill reuse while preserving verifier-critical evidence.The framework is evaluated across synthesis, evolution, retrieval reuse, and trajectory fine-tuning, with watermarking providing empirical provenance signals during trace reuse.

Limitations

The study evaluates black-box trace disclosure on controlled long-horizon tasks, so its findings may not fully generalize to noisier deployments with changing tools and weaker verifier feedback. It also finds contextual watermarks harder to retain than standalone hooks, despite low false alarms.

  • Evaluation scope: The evaluation covers black-box trace disclosure, excluding private skill files and model internals from downstream users.The benchmark uses released traces observed without access to those private components.
  • Evaluation scope: The evaluation uses 75 controlled long-horizon tasks with automatic verifiers, which may not represent noisier real deployments.The passage identifies changing tool environments and weaker verifier feedback as additional deployment differences.
  • Watermark retention: Contextual watermarks are harder to retain than standalone hooks, although they maintain low false alarms.The passage reports this contrast in the provenance results without specifying a false-alarm value.

Ethics Statement · A Benchmark and Dataset Details

The paper frames safer agent-trace release as a protection-oriented effort against proprietary and security-sensitive procedure leakage. It introduces CAPTRACEBENCH as a benchmark using controlled tasks, automatic verifiers, and verifier families for evaluation.

  • Ethics Statement: The work targets safer release of agent traces containing proprietary or security-sensitive procedures.
  • Ethics Statement: The benchmark uses controlled tasks with automatic verifiers.
  • Ethics Statement: The proposed release policy preserves audit evidence while reducing capability leakage.
  • Ethics Statement: The analysis explicitly states its threat model and reports results in a protection-oriented setting.
  • A Benchmark and Dataset Details: CAPTRACEBENCH construction covers task and skill statistics and verifier families for automatic evaluation.
  • A Benchmark and Dataset Details: The section describes verifier families used for automatic evaluation.

A.1 Benchmark Construction … C Experiment Details

The paper evaluates RedAct on CAPTRACEBENCH, a 75-task benchmark with curated skills and automatic verifiers, and details protected trace release plus behavioral watermarking. The methods preserve verifier-critical evidence while abstracting protected procedural information and define provenance checks with explicit controls and failure modes.

  • A.1 Benchmark Construction: CAPTRACEBENCH contains 75 long-horizon tasks spanning 23 task families and 7 domains, with 154 curated skill files and 3,783 successful teacher trajectories.Each benchmark task includes an instruction, local SKILL.md files, and an automatic verifier.
  • A.2 Verifier Specifications: Task-local verifiers assess structured outputs, numeric answers, repaired code and formal methods, generated artifacts, and domain-specific requirements, determining correctness and step-level progress.Supported checks include JSON, YAML, and CSV schema validation; exact or tolerance-based numeric checks; unit tests; artifact checks; and validation scripts.
  • B Method Details; B.1 Protected Release Pipeline; B.1.1 Protected Trace Release Algorithm: The protected release algorithm takes a task instruction, private skill package, and teacher trajectory, extracts protected key information, rewrites assistant turns, and rejects outputs that fail format or final-answer preservation.The release process retains verifier-critical evidence while generalizing protected procedural details.
  • B.1.2 Key-Information Extraction and Rewriting Prompts: The extraction and rewriting prompts abstract protected formulas, parameters, tool choices, code details, and intermediate values while preserving task goals, turn counts, tool-use evidence, execution order, final answers, and verifier-critical fields.The prompt templates are collected in Appendix E.
  • B.2 Behavioral Watermarking; B.2.1 Watermark Injection Algorithm; B.2.2 Watermark Families and Templates: RedAct’s watermarking uses four natural behavioral families: standalone Ritual Marker and Env Check hooks, plus tool- or error-anchored Cross Check and Error Anchoring hooks.Cross Check produces a tool-grounded sentence under 25 words, while Error Anchoring inserts a recovery statement after a rare realization phrase.
  • B.2.3 Provenance Detection Protocol; C Experiment Details: Provenance detection compares true detection and false alarm rates between matching triggered evaluations and control evaluations using matching activation keys.The positive condition uses a matching watermarked student, whereas the control uses the base model under the same key.
  • B.2.4 Detection Controls and Failure Modes: Provenance tests require positive and negative controls covering unwatermarked students, missing activation keys, and mismatched keys.These controls distinguish matching watermark behavior from alternative or absent conditions.
  • B.2.4 Detection Controls and Failure Modes: Watermark provenance signals can weaken when adversaries remove fixed markers, strip tool-result wrappers, paraphrase rare recovery phrases, or continue training the student model.The passage identifies these transformations as failure modes for provenance detection.

C.1 Metric Definitions … D.2.2 Human Rating

The paper defines metrics for correctness, procedural progress, skill transfer, and protected-information recovery, then evaluates trace reuse under controlled disclosure conditions. Supplementary analyses describe implementation settings and assess trace quality through perplexity and human ratings.

  • C.1 Metric Definitions: Run-level evaluation records submitted outputs, verifier correctness, and stepwise procedural predicates across task checks.For each valid run, the submitted output, task verifier, and k-th step predicate among K_t checks are defined.
  • C.1 Metric Definitions: Normalized Skill Transfer compares mean SSR across none, original, raw, and protected conditions, while RPI counts explicitly recovered protected key items.Conservative text matching accepts explicit items or close lexical variants and treats lower NST and RPI as less recovered protected procedure.
  • C.2 Downstream Reuse Methods: Downstream reuse methods receive only public artifacts and must reconstruct executable procedures without hidden prompts, skills, weights, or internal states.The evaluation is same-task and in-distribution, sharing task instructions and verifiers while requiring reconstruction from public traces alone.
  • C.2 Downstream Reuse Methods: The benchmark tests single-agent synthesis, multi-agent evolution, and retrieval reuse as distinct ways to reuse released trajectories.Synthesis induces a SKILL.md and optional scripts; evolution revises induced skills over multiple passes; retrieval supplies top-k snippets before solving the task.
  • C.3 Closed-Source Models and Agent Harnesses: Closed-source agents run through the same batch runner in isolated Docker task environments built from ubuntu:24.04 with task-specific dependencies.Model and harness identifiers follow public provider documentation and are listed in Table 9.
  • C.3 Closed-Source Models and Agent Harnesses: Deterministic or near-deterministic calls use temperature 0, while watermark hook placement and detection are rule-based after auxiliary-LLM hook generation.This implementation applies to key-information extraction, trajectory rewriting, Cross Check, and Error Anchoring.
  • C.4 Open-Model Fine-Tuning Setup: LlamaFactory supports multi-turn supervised fine-tuning, retaining 2,033 watermarked trajectories under 32k tokens for Qwen3-8B/4B provenance experiments.Table 10 summarizes the associated trajectory fine-tuning hyperparameters.
  • D Additional Experiment Results; D.1 Per-Difficulty Results; D.2 Trace Quality Analysis; D.2.1 Perplexity; D.2.2 Human Rating: Supplementary results cover per-difficulty SSR and trace-quality diagnostics, including perplexity, naturalness, usability, agreement, and human rubric ratings.Protected traces have modestly higher PPL than raw traces, while two annotators rate 50 protected traces for naturalness and usability using quadratic weighted Cohen’s κ.

D.3 Provenance Detection Results … Agent Trajectory after RedAct Protection

The merged sections present provenance-detection materials, prompt templates, and a lake-warming case study showing that RedAct can protect procedural details while retaining workflow execution and verifier-relevant outputs.

  • E Prompt Templates: The protected trace release uses a key-information extraction prompt to identify sensitive professional information from task instructions and skill documents.The prompt is presented as a professional key information identification assistant.
  • D.3 Provenance Detection Results: Table 12 reports Qwen3-8B provenance detection across watermark ratios, while Figure 9 compares token-length and tool-call distributions for ratio-0.3 watermarked trajectories.TD and FA are reported in percentages.
  • E Prompt Templates: The trajectory rewriting prompt generalizes sensitive terms, formulas, algorithms, library calls, parameters, intermediate values, and implementation identifiers while preserving completeness and utility.Sensitive code retains an executable wrapper and basic file I/O but replaces protected logic with a runnable generic skeleton.
  • E Prompt Templates: The contextual cross-check prompt generates one constrained verification sentence after a tool result to support assistant-trajectory watermarking.It uses the tool call and result so the verification sounds relevant.
  • F Case Study: The case study shows RedAct protecting a released trace without reducing it to an answer-only summary, despite the raw trajectory exposing reusable schemas, skill names, methods, scripts, paths, and numerical results.The exposed workflow includes Mann-Kendall trend testing, Sen’s slope, Factor Analysis, driver groups, intermediate scripts, output paths, and final numerical results.
  • Agent Trajectory before RedAct Protection: The raw lake-warming trajectory explicitly reveals file schemas, skill definitions, Mann-Kendall and Sen’s-slope procedures, Factor Analysis, category mappings, library checks, scripts, and generated outputs.The workflow maps variables to Heat, Flow, Wind, and Human and computes category contribution by R2 decomposition.
  • Agent Trajectory before RedAct Protection: 0.08786727179630346 slope and 0.03433985000756268 p-value were produced, while Heat contributed 53.208173224976385 in the verified CSV outputs.The final response describes a statistically significant warming trend and approximately 53.21% Heat contribution.
  • Agent Trajectory after RedAct Protection: After RedAct protection, the trajectory retains environment checks, data inspection, script execution, CSV verification, and the same analytical outputs while presenting less procedural detail.The protected trajectory reports the same slope, p-value, and Heat contribution, and marks an injected environment-probing action as provenance evidence.
Loading 2606.10813v3…