PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentLeak: Cloning Stronger LLM Agent Capabilities onto Weaker Agents Beyond Skill Stealing

Xiaoting Lyu, Yuhong Wu, Yufei Han, Shichang Liu, Liang Zhang, Bin Wang, Bin Wang, Xiaobo Ma, Wei Wang

arXiv:2609.07131v1cs.CRcs.AI

TL;DR

Prior extraction work recovers agent artifacts, but whether stolen skills transfer a stronger proprietary agent’s capability to a substantially weaker agent remains an open question. AgentLeak uses observable victim–attacker execution differences to add missing procedural behaviors to attacker-side skills while leaving the attacker’s model, harness, and tools unchanged. It improves pass rates over direct skill reuse and recovers over 80% of the capability gap.

  • Problem

    Prior work extracts skills and other agent artifacts, but whether stolen skills transfer task-solving capability to a substantially weaker agent remains unexamined.

    Source: §1 · pp. 1–2

  • Method

    AgentLeak diagnoses missing procedural behaviors from victim–attacker execution differences and encodes them into attacker-side skills while keeping the attacker’s model, harness, and tools fixed.

    Source: §1 · p. 2

  • Results

    AgentLeak improves task pass rates by over 40 percentage points compared with direct skill reuse and recovers over 80% of the victim–attacker capability gap.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Observable execution behavior can leak procedural information that enables reconstruction of proprietary task-solving capability beyond explicit artifact leakage.

    Source: §7 · p. 14

    AgentLeak is less applicable when victim agents expose only final outputs or highly restricted interfaces, and cannot reconstruct capabilities dependent on inaccessible resources or model-internal knowledge.

    Source: §6.2 · p. 13

Abstract

from arXiv · show

Large language model (LLM) agents increasingly achieve long-horizon tasks by combining foundation models with explicit skills and implicit procedural knowledge acquired through execution. The resulting task-solving capabilities have become valuable proprietary assets, raising a new security question: can a substantially weaker attacker-controlled agent acquire the capabilities of a stronger proprietary agent through limited black-box interaction? Existing skill-stealing attacks recover explicit skill artifacts, yet we show that artifact leakage does not necessarily transfer capability: a weaker agent may possess the same skills but still fail because it lacks procedural behaviors implicitly realized by the stronger agent. Our key insight is that the skill execution gap itself forms a leakage surface, where missing behaviors are exposed through observable differences between successful victim executions and failed attacker executions. Based on this, we present AgentLeak, a black-box capability-cloning attack that identifies capability-critical behaviors from these execution differences and incorporates them into attacker-side skills, while keeping the attacker's model, harness, and tools unchanged. Across 20 task scenarios comprising 600 instances, diverse agent systems, and multiple backbone models, AgentLeak improves task pass rates by over 40% compared with direct skill reuse and recovers more than 80% of the victim--attacker capability gap. Our findings reveal a confidentiality risk in LLM agents: protecting explicit artifacts alone is insufficient, as observable execution behavior can leak the procedural knowledge required to reconstruct proprietary task-solving capabilities in low-capability and attacker-controlled agents.

1 Introduction

The introduction asks whether stolen skills can transfer a stronger agent’s task-solving capability to a substantially weaker one, and identifies observable execution differences as a way to recover missing procedural behaviors. AgentLeak uses those differences to revise attacker-side skills, reporting substantial capability recovery across diverse tasks and systems.

1 Introduction

  • Prior work extracts models, prompts, or skills, but leaves open whether stolen skills transfer task-solving capability to a substantially weaker agent.The introduction frames this as the question of whether a weaker attacker can gain a low-cost substitute for a proprietary service.Source: §1 · p. 1
  • Directly acquiring a stronger agent’s skills gives a weaker agent only limited improvement, and skill optimization still fails to close the capability gap.The motivating study compares a Codex agent using GPT-5.5 with an OpenHands attacker using Qwen3.6-35B.Source: §1 · p. 2, Figure 1 · p. 2
  • Skills specify procedures, but stronger agents can supply implicit execution behaviors—such as decomposition, tool selection, feedback interpretation, and verification—that weaker agents lack.Thus, identical skill instructions need not yield identical end-to-end capability.Source: §1 · p. 2
  • The threat model gives the attacker limited black-box queries and observable execution traces, while restricting changes to local skills and keeping its model, harness, and tools unchanged.The attacker does not access victim-private skills, model parameters, hidden reasoning, memory, or other privileged information.Source: §1 · p. 2
  • AgentLeak abstracts successful victim trajectories into workflows, diagnoses capability-critical deficits through victim–attacker comparison, and rewrites attacker skills with reusable behavioral primitives.It iteratively reruns refined skills and validates them on previously solved tasks while keeping the attacker’s model fixed.Source: §1 · p. 2
  • Over 40 percentage points higher task pass rates than direct skill reuse and skill evolution, with over 80% of the victim–attacker capability gap recovered.The evaluation covers 20 task scenarios and 600 instances, using actual executions in native task environments.Source: §1 · pp. 2–3

2 Background and Related Work

This section distinguishes explicit skills from the execution behaviors and end-to-end capability they help produce. It contrasts self-directed skill refinement and artifact-extraction attacks with AgentLeak’s adversarial transfer of capability from a stronger agent to a weaker one.

2.1 Agent Skills, Execution, and Capability

  • Skills encode explicit procedural knowledge, but capability also depends on execution behaviors, tools, and the environment, so identical skills can yield different task outcomes.Execution includes planning, tool interactions, feedback handling, and verification.Source: §2.1 · p. 3

2.2 Skill Evolution and Refinement

  • Skill-evolution methods refine skills from an agent’s own execution experience, whereas AgentLeak targets adversarial transfer from a stronger proprietary agent to a weaker attacker-controlled one.The section names Trace2Skill, SkillX, SkillGen, SkillAdaptor, and SkillOpt as examples of skill-generation or refinement methods.Source: §2.2 · p. 3

2.3 Extraction Attacks on LLM Agents

  • Extraction attacks target models, prompts, or skill artifacts, while AgentLeak targets transfer of a stronger agent’s task-solving capability to a weaker agent.Prior skill-extraction work recovers content, infers skills from trajectories, or reconstructs hidden skill functionality.Source: §2.3 · pp. 3–4

3 Problem Formulation

The paper models skill-augmented agents and their task executions, then formalizes capability cloning as transferring a stronger proprietary agent’s task-solving capability to a weaker agent by modifying only its skills. The attacker has limited black-box access and no privileged victim information.

3.1 System Model

  • An agent is modeled as A = (M,H,U), comprising an underlying LLM, an orchestration and execution-control harness, and available external tools.Source: §3.1 · p. 4
  • Each task instance specifies an instruction, initial execution environment, acceptance criteria, and a verifier that determines whether execution meets those criteria.Source: §3.1 · p. 4
  • Agent execution produces an observable trajectory of actions and observations, while a verifier determines whether the task succeeds against its acceptance criteria.Source: §3.1 · p. 4
  • Task-solving capability is defined through an agent’s expected task success rate on a scenario, providing the basis for capability-transfer and capability-gap measurements.Source: §3.1 · p. 4

3.2 Threat Model

  • The attacker controls a substantially weaker open-weight agent and may change only its skills; its model, harness, and tools remain fixed during reconstruction.Source: §3.2 · p. 4
  • The objective is to transfer the victim’s realized task-solving capability—not merely recover its artifacts—using skill optimization so the weaker agent can generalize to unseen instances and work without further victim access.Source: §3.2 · p. 4
  • The attacker may query the victim on limited instances and observe normal-interface execution evidence, but cannot access private skills, parameters, prompts, hidden states, memory, or unexposed reasoning.Source: §3.2 · pp. 4–5
  • Initial attacker skills can come from public sources, extraction techniques, or attacker generation, and the attacker knows the target scenario and its task instances.Source: §3.2 · p. 5

4 AgentLeak: Capability Cloning Attack

AgentLeak reconstructs a stronger agent’s capability by turning execution differences into procedural guidance in the weaker agent’s skills. It abstracts reusable workflows, diagnoses capability deficits, extracts behavioral primitives, and applies localized skill updates iteratively.

4.1 Attack Overview

  • AgentLeak uses limited victim execution evidence to identify missing procedural behaviors and refine attacker-side skills while keeping the model, harness, and tools unchanged.Source: §4.1 · p. 5

4.2.1 Workflow Abstraction and Skill Mapping

  • Victim trajectories are abstracted into reusable workflow skills that structure execution and coordinate existing skills, rather than being copied as instance-specific traces.Source: §4.2.1 · p. 6

4.2.2 Differential Capability-Deficit Diagnosis

  • Workflow-guided diagnosis separates capability-critical deficits from benign execution variations and prioritizes recurring failure-causing behaviors for reconstruction.Source: §4.2.2 · pp. 6–7

4.2.3 Behavioral Primitive Extraction

  • Behavioral primitives translate diagnosed deficits into reusable procedural improvements, such as adding artifact validation and recovery rather than copying task-specific commands.Source: §4.2.3 · p. 7

4.2.4 Structured Skill Rewriting

  • Structured rewriting converts primitives into localized Insert or Replace patches, preserving effective skill components while addressing identified deficits.Source: §4.2.4 · p. 7

4.3 End-to-End Attack Procedure

  • After processing query instances, AgentLeak checks for regressions on previously solved tasks and freezes the resulting skill set as the capability clone.Source: §4.2.4 · p. 8, §4.3 · p. 8
  • For each failed query instance, the end-to-end procedure obtains a successful victim trajectory, builds a workflow, diagnoses deficits, extracts primitives, and iterates updates until success or budget exhaustion.Source: §4.3 · p. 8

5 Evaluation

The evaluation tests whether AgentLeak reconstructs capabilities across tasks, agent configurations, transfer settings, and defenses. It reports stronger capability recovery than skill-reuse baselines, while measuring component contributions and practical attack conditions.

5.1 Experimental Setup

  • Evaluation uses 20 SkillsBench scenarios expanded to 600 instances, with one query instance and 29 held-out instances per scenario.Victim systems include proprietary products and general-purpose frameworks; weaker attackers use OpenHands with Qwen 3.6-35B-A3B or Gemma 4-31B.Source: §5.1 · pp. 8–9

5.2.1 Overall Attack Performance

  • Raw Skill and Trace2Skill leave substantial gaps; Trace2Skill reaches only 34.96% IPR for Qwen and 28.81% IPR for Gemma.Raw Skill reaches 30.64% IPR for Qwen and 35.59% IPR for Gemma, while recovering only 17.38%–19.61% and 32.48%–37.77% CRR, respectively.Source: §5.2.1 · p. 10
  • AgentLeak achieves over 70% IPR across all victim–attacker configurations, outperforming Raw Skill and Trace2Skill by more than 35 percentage points, demonstrating transfer of stronger victims’ task-solving capabilities to substantially weaker attackers.Source: §5.2.1 · p. 10

5.2.2 Transferability of Reconstructed Skills

  • Frozen reconstructed skills transfer across models and harnesses, reaching 72.23% IPR and 86.16% CRR cross-model, and 83.33% IPR and 104.87% CRR across both.In cross-harness transfer, IPR rises from 70.06% to 84.68% and CRR from 82.50% to 107.15%.Source: §5.2.2 · p. 11, Table 5 · p. 11

5.2.3 Ablation Study

  • Removing workflow abstraction drops IPR from 77.80% to 22.33% and CRR from 95.50% to 2.00%; removing the other components also degrades recovery.The ablations cover differential diagnosis, behavioral primitives, and structured skill rewriting.Source: §5.2.3 · pp. 11–12

5.2.4 Attack Feasibility Analysis

  • After two rounds, Qwen CRR reaches around 90% and saturates; across skill-complexity levels, CRR stays above 84% for Qwen and around 90% for Gemma.Per scenario, reconstruction uses one victim query and averages 6.34 attacker executions, 7.34 auxiliary-model calls, $6.32, and 60 minutes.Source: §5.2.4 · p. 12, Figure 4 · p. 12, Figure 5 · p. 12

5.3 Defense Analysis

  • Trajectory redaction reduces CRR from 95.44% to 73.23% and IPR to 75.21%, a larger degradation than artifact-level defenses.Artifact-level defenses reduce CRR by only 7.1–18.2 percentage points; redaction still leaves substantial capability recovery.Source: Table 7 · p. 12, §5.3 · p. 13

6 Discussion

AgentLeak shows that LLM-agent capabilities can leak through observable execution behavior even when internal components remain protected. The attack depends on sufficient interaction feedback and cannot reconstruct capabilities requiring inaccessible resources.

6.1 Capability Leakage Risks

  • Capability confidentiality extends beyond protecting model parameters, prompts, and skills because procedural behaviors can be reconstructed from observable execution.The paper identifies action selection, tool invocation, verification, and recovery strategies as behavioral leakage surfaces.Source: §6.1 · p. 13
  • AgentLeak infers capability-critical behaviors from black-box executions and adds them to weaker-agent skills without reconstructing the proprietary agent system.The attacker’s model, harness, and tools remain unchanged.Source: §6.1 · p. 13

6.2 Limitations

  • The attack requires sufficient victim-interaction feedback, is less applicable to final-output-only or restricted interfaces, and cannot recover capabilities dependent on inaccessible resources.Examples include private data, victim-exclusive tools, proprietary services, and model-internal knowledge.Source: §6.2 · p. 13

7 Conclusion

The paper identifies capability leakage as a risk when weaker agents reconstruct stronger agents’ task-solving capabilities from observable executions. AgentLeak extracts capability-critical behaviors from execution differences and incorporates them into attacker-side skills, while leaving the attacker’s model, harness, and tools unchanged.

7 Conclusion

  • AgentLeak reconstructs a stronger agent’s task-solving capability from limited observable executions without replicating its underlying system.The conclusion frames this as a capability-leakage risk.Source: §7 · p. 13
  • AgentLeak’s evaluations across diverse task scenarios, agent frameworks, and backbone models show substantial improvement in weaker agents and recovery of a large fraction of the victim–attacker capability gap.The approach encodes capability-critical behaviors inferred from execution differences into attacker-side skills while keeping the underlying model, harness, and tools unchanged.Source: §7 · pp. 13–14
  • The findings indicate that protecting proprietary agents requires attention to behavioral information exposed during execution, not only explicit artifacts.The attack leaves the attacker’s model, harness, and tools unchanged.Source: §7 · p. 14

Ethical Considerations

The work characterizes capability-cloning risks using controlled research experiments and authorized agent configurations. Its appendix describes LLM-assistant prompts for turning victim–attacker execution differences into reusable workflow guidance, diagnoses, behavioral primitives, and skill rewrites.

Ethical Considerations

  • Experiments use publicly available benchmarks and authorized agent configurations, without accessing private user data or confidential information.Collected traces are used solely for experimental analysis, and the stated goal is to characterize risks and motivate defenses.Source: Ethical Considerations · p. 15

A Prompts Used by the LLM Assistant

  • The LLM assistant is used offline to abstract workflows, diagnose capability deficits, extract behavioral primitives, and rewrite skills while leaving the attacker model, framework, and tools unchanged.The prompts turn victim–attacker execution differences into reusable procedural knowledge.Source: A Prompts Used by the LLM Assistant · p. 17
  • Workflow abstraction divides successful trajectories into behavioral segments, then removes run-specific details while retaining reusable procedures, dependencies, validation, and recovery strategies.Steps are associated with skills and relevant sections; cross-skill coordination and workflow understanding map to the workflow skill.Source: A Prompts Used by the LLM Assistant · p. 17
  • Capability-deficit diagnosis retains failure-contributing behavioral gaps, routes each to a responsible skill section, and distinguishes new gaps from previously patched but unresolved ones.The workflow prompt merges matching diagnoses and preserves unmatched records.Source: A Prompts Used by the LLM Assistant · pp. 17–18
  • Primitive extraction ranks diagnoses and converts the leading ones into executable, observable, verifiable requirements; structured rewriting groups them into minimal skill-section edits.The prompts limit extraction to the first three diagnoses and at most five primitives, and limit rewriting to at most five edits.Source: A Prompts Used by the LLM Assistant · p. 18
  • The rewrite prompt changes guidance according to reconstruction status and avoids hard-coding run-specific content or changing the task contract.It may add missing procedure guidance or, when that remains insufficient, add an Execution subsection referenced from Procedure.Source: A Prompts Used by the LLM Assistant · p. 18

B Example of a Reconstructed Skill

The appendix example illustrates how AgentLeak turns victim–attacker execution differences into targeted skill refinements.

B Example of a Reconstructed Skill

Original Skill

The original skill outlines a business-cycle analysis workflow: prepare and transform economic series, detrend them, then analyze their cyclical components.

  • The process loads and cleans data, converts nominal values to real terms, and log-transforms positive level variables.It also handles missing values and ensures proper time ordering.Source: Original Skill · p. 18
  • It applies an HP filter with a smoothing parameter appropriate to the data frequency.Source: Original Skill · p. 19
  • The resulting cyclical components are analyzed using statistics such as correlations and volatilities.The passage includes a business-cycle correlation example.Source: Original Skill · p. 19

Reconstructed Skill

The reconstructed skill specifies a validated workflow for turning nominal series into aligned cyclical components and calculating business-cycle statistics.

  • Quarterly continuation labels inherit the most recently stated year, and annual values without supplied observations are calculated from every available quarter in that year.The procedure checks that each quarter appears exactly once and that annual means use the full group.Source: Reconstructed Skill · p. 19
  • Nominal series are aligned with the appropriate price index by year and converted to real terms.A normalized price ratio requires joining each nominal observation to the ratio for the same year.Source: Reconstructed Skill · p. 19
  • Only finite, strictly positive values are logged before applying an HP filter with a frequency-appropriate smoothing parameter.Source: Reconstructed Skill · p. 19
  • The cyclical components are aligned on common years before computing correlations, volatility, or another requested business-cycle statistic.The final inspection checks that intended years appear once per series, values are present and finite, and the statistic is finite.Source: Reconstructed Skill · p. 19
  • When a normalized price ratio is misinterpreted or joined incorrectly, the procedure calls for correcting it and recomputing the complete real series.Source: Reconstructed Skill · p. 19

C Task Selection and Instance Construction

The evaluation selects varied, multi-step task scenarios and constructs validated instances that vary inputs while preserving each task's core capability requirements.

Task selection

  • Scenarios span application domains, difficulty levels, and skill compositions, prioritizing multi-step execution and specialized procedural knowledge.Variations within a task family must preserve its objective, workflow, tool use, skill requirements, and evaluation semantics.Source: C Task Selection and Instance… · p. 19

Task instance construction

  • Each selected scenario receives 30 instances with controlled changes to inputs, targets, parameters, files, or application contexts.The original raw skills, workflow, tool interfaces, and evaluation criteria remain unchanged; variations that change required capabilities or introduce new stages or tools are excluded.Source: C Task Selection and Instance… · p. 19

Instance validation

  • Generated instances undergo LLM-based validation and manual review for task-family fit, intended variation, consistency, and executability.Instances with unintended structural changes or task drift are revised or discarded.Source: C Task Selection and Instance… · p. 19

Instance examples

  • Each task package typically includes instructions, environment inputs, skills, a reference solution or oracle, and a test-based verifier.Only files or task attributes associated with designated variation dimensions are modified.Source: C Task Selection and Instance… · p. 20
  • The examples cover PPTX reference formatting and PDF–Excel difference detection, with generalization directions summarized in Tables 8 and 9.SS varies one dimension, while MS varies two compatible dimensions simultaneously.Source: C Task Selection and Instance… · p. 20, Table 8 · p. 20, Table 9 · p. 20

D Detailed Scenario-level Results

The appendix reports scenario-level AgentLeak performance, complementing the main paper's aggregated evaluation.

Loading 2609.07131v1…