PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, Jun Wang

arXiv:2602.01869v3cs.AI

TL;DR

LLM agents often re-derive solutions in recurring situations, limiting experience reuse and increasing computational redundancy. Skill-Pro learns executable procedural Skills without parameter updates, using a Skill-MDP and Non-Parametric PPO. Across diverse scenarios, it achieves superior reuse rates and performance gains with extreme memory compression.

  • Problem

    LLM agents commonly redo full reasoning for recurring situations, creating computational redundancy and increasing error accumulation risks in long-horizon execution.

  • Method

    Skill-Pro formalizes executable Skills in a Skill-MDP and evolves them without parameter updates using semantic gradients, a PPO Gate, and score-based maintenance.

  • Results

    Skill-Pro achieves superior reuse rates and significant performance gains with extreme memory compression across in-domain, cross-task, and cross-agent scenarios.

  • Takeaways & Limitations

    High-quality procedural memory is more efficient than raw episodic storage for long-term autonomy within the evaluated scenarios.

  • Takeaways & Limitations

    The current Skill-selection mechanisms are simple instantiations, and semantic-gradient updates may extrapolate beyond their feedback.

Abstract

from arXiv · show

LLM-driven agents excel at sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and instability. To bridge this gap, we propose Skill-Pro, a framework enabling agents to autonomously learn reusable procedural skills from interaction experiences without parameter updates. By formalizing a Skill-MDP, Skill-Pro transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, Skill-Pro sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that Skill-Pro achieves superior reuse rates and significant gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how Skill-Pro transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.

1. Introduction

LLM agents often re-derive solutions in recurring situations, creating computational redundancy and reliability risks. Skill-Pro addresses this gap by learning executable procedural Skills without parameter updates and reports strong reuse and performance with compressed memory.

  • Episodic memory retrieves past interactions for reference, whereas procedural memory directly maps situations to reusable action patterns.
  • Recurring situations still trigger full reasoning from scratch, increasing computational redundancy and long-horizon error accumulation.
  • Skill-Pro formalizes Skills with activation, execution, and termination conditions within a Skill-MDP, converting passive narratives into executable procedures.
  • Non-Parametric PPO generates refined Skill candidates from semantic gradients and verifies them through a PPO-style Trust-Region Verification gate.
  • Skill-Pro reports superior reuse rates, performance gains, and extreme memory compression across in-domain, cross-task, and cross-agent scenarios.

2. Related Work

Prior work adapts LLM agents through parameter updates or external memory. These approaches differ in their costs, risks, and representations of interaction experience.

  • Parametric adaptation can improve performance but incurs high computational costs and risks catastrophic forgetting or overspecialization.
  • Memory-augmented agents provide a non-parametric alternative by storing external experience and retrieving it to condition decision-time reasoning.
  • Existing memory systems represent experience as episodic trajectories, abstracted knowledge, or structured and compressed memory.

3. Reusable Procedural Units: Skills

Skill-Pro models reusable procedural units as temporally extended Skills and organizes their selection, execution, and evolution in a Skill-MDP. The framework keeps the LLM and selector fixed while evolving the Skill pool from interaction trajectories.

  • 3. Reusable Procedural Units: Skills: Skills are reusable procedural units specifying activation conditions, execution procedures, and termination conditions.
  • 3.1. Problem Formulation: The Skill-MDP extends an MDP with a dynamic Skill pool Ω that represents procedural memory and organizes reusable Skill selection and execution.
  • 3.1. Problem Formulation: The hierarchical policy factorizes into Skill selection conditioned on state and pool, followed by LLM-generated primitive actions conditioned on state and the active Skill.
  • 3.2. Skill Structure: A Skill’s natural-language activation condition identifies applicable observable contexts, while its execution procedure supplies ordered actions without re-deriving reasoning.
  • 3.2. Skill Structure: A termination condition ends Skill execution when satisfied; otherwise the Skill remains active and control stays with it.
  • 3.3. Skill Selection: Skill selection supports similarity-based retrieval and value-based prioritization among the most similar Skills, while more advanced retrieval policies remain possible.
  • 3.4. Skill Pool Evolution: The Skill pool evolves from collected trajectories by synthesizing new Skills, refining existing ones, and pruning empirically underperforming Skills.
  • 3.4. Skill Pool Evolution: The framework focuses on evolving the Skill pool while keeping the LLM action policy and Skill-selection policy fixed.

4. Non-Parametric PPO for Skill Evolution

Non-Parametric PPO evolves reusable Skills without updating LLM parameters by generating semantic-gradient candidates and verifying them with a PPO-style trust region. Score-based maintenance then retains high-contribution Skills within a fixed-capacity pool.

  • Non-Parametric PPO evolves the Skill pool without updating any LLM parameters.It replaces parameter updates with Skill refinement under PPO-style trust-region principles.
  • Semantic Gradients: Semantic Gradients use hindsight feedback from interaction trajectories to specify refinements to a Skill’s activation, execution, and termination conditions.Batch aggregation extracts recurring failure patterns and consistent suggestions while filtering conflicting or trajectory-specific signals.
  • PPO-Style Trust-Region Verification: Semantic-gradient candidates may extrapolate beyond observed interactions and introduce hallucinated or behaviorally unstable Skills.The PPO-style verification step is introduced to mitigate this risk before candidates enter the Skill pool.
  • PPO-Style Trust-Region Verification: The PPO Gate verifies candidate Skills by favoring higher probability for high-advantage historical actions while limiting deviations from the behavior policy.The frozen LLM supplies the policy, and advantages are estimated from return-to-go with a running baseline rather than a trained value function.
  • PPO-Style Trust-Region Verification: Best-of-Nc selection chooses the highest-scoring candidate and accepts it only when its PPO Gate score is positive.This filters unreliable or hallucinated candidates under the trust-region constraint.
  • Score-Based Skill Maintenance: Online Skill scores measure average advantage during invocation and support fixed-capacity maintenance through removal of non-positive, duplicate, redundant, or lowest-scoring Skills.The score uses cumulative gain divided by invocation count, with a minimum denominator of one.

5. Experiments

Skill-Pro is evaluated on reuse, efficiency, performance, component contributions, and procedural-memory evolution across tasks and agents. It consistently outperforms baselines in reuse while maintaining a highly compressed memory footprint and strong performance.

  • 5.2. Does Skill-Pro Truly Enable Reusability?: Skill-Pro consistently outperforms all baselines in in-domain, cross-task, and cross-agent memory reuse rates.Results average 50 episodes per setting, with memories built on in-domain tasks and reused across out-of-distribution, higher-difficulty, or heterogeneous-agent settings.
  • 5.2. Does Skill-Pro Truly Enable Reusability?: 816 tokens is Skill-Pro’s maintained memory size, substantially below the hundreds of thousands of tokens accumulated by episodic baselines.Its compact representation also reduces prompt bloat, execution load, and per-step retrieval through temporally extended Skills.
  • 5.2. Does Skill-Pro Truly Enable Reusability?: 0.90 is Skill-Pro’s reported highest success rate in ALFWorld despite extreme memory compression.The learned memory also yields performance gains across varying task difficulties and LLM backbones.
  • 5.3. Why Does Skill-Pro Work?: 0.606 → 0.388 is the performance drop for the w/o Skill variant, showing that procedural Skills are essential building blocks for complex decision-making.Removing NP-PPO also prevents general seeds from being refined into task-specific expertise, reducing reuse and success rates.
  • 5.3. Why Does Skill-Pro Work?: 30% is the drop in PPO Gate Pass Rate when Semantic Gradients are removed, while removing the PPO Gate destabilizes training by admitting all candidates.The w/o SG variant remains more stable because its candidates still undergo trust-region verification.
  • 5.4. How Does Skill-Pro Evolve and Reuse?: −0.0018 is the w/o Score Online Score, indicating that FIFO replacement discards high-performing Skills for unproven newcomers.Figure 4 further shows iterative refinement and score-based pruning supporting a compact, high-utility Skill pool.
  • 5.4. How Does Skill-Pro Evolve and Reuse?: Skill selection patterns remain stable across task difficulties, while backbone-specific profiles differ, including greater FBInference reliance for Gemma2-9B.StratPlan maintains consistent activation across agents and functions as a standardized procedural primitive.

6. Conclusion

Skill-Pro learns executable procedural Skills without parameter updates by combining Skill-MDP formalization with Non-Parametric PPO and score-based maintenance. Across diverse scenarios, it achieves superior reuse and performance with extreme memory compression.

  • 6. Conclusion: Skill-Pro formalizes passive episodic narratives as executable, reusable Skills through the Skill-MDP.Each Skill includes activation, execution, and termination conditions for decision-making.
  • 6. Conclusion: Non-Parametric PPO generates candidates with semantic gradients and verifies them with a PPO Gate under a frozen LLM policy.Score-based maintenance then prunes low-return Skills to sustain long-term memory quality.
  • 6. Conclusion: Skill-Pro achieves superior reuse rates and significant performance gains with extreme memory compression across diverse scenarios.The conclusion characterizes procedural memory as more efficient than raw episodic storage for long-term autonomy.

Impact Statement

The paper aims to improve the efficiency and reusability of autonomous agents by enabling procedural knowledge accumulation without continuous parameter updates.

  • Impact Statement: Skill-Pro targets more efficient and reusable autonomous agents through procedural knowledge accumulation without continuous parameter updates.

Summary of Appendices

The appendices provide related-work context, experimental details, case studies, prompt templates, and discussion of limitations. They also describe the benchmarks, baselines, and backbone configurations used to assess Skill-Pro.

  • Appendix overview: Sections A–C cover full related works, detailed experimental setup, and additional experimental details.
  • Appendix overview: Sections D–F provide a case study, prompt templates, and discussion of limitations.
  • Related work: The related work positions Skill-Pro among parametric and non-parametric experience-learning methods, including memory-augmented and procedural-memory agents.
  • Experimental setup: TextArena and ALFWorld evaluate repeated experience reuse in text-based games and embodied environments, respectively.
  • Experimental setup: Baselines include trajectory retrieval, distilled insights, concise notes, structured workflows, hybrid memory, and reasoning-based agents.
  • Experimental setup: TextArena experiments reuse Gemma-2-9B Skills across Gemma-3-4B, Qwen3-32B, and LLaMA-3.3-70B-Instruct, while ALFWorld uses Qwen3-32B.

B.4. Evaluation Metrics

The evaluation measures task performance, experience reuse, and efficiency across storage and inference costs. Reuse is assessed within domains, across tasks, and across LLM backbones.

  • Reuse Metrics: Experience reuse is measured through in-domain, cross-task, and cross-agent reuse rates.These metrics quantify reuse within a task domain, generalization across tasks, and reuse across different LLM backbones.
  • Task Performance: Task performance measures sequential decision-making ability using benchmark-specific metrics.TextArena uses average return per episode, while ALFWorld uses success rate.
  • Efficiency Metrics: Efficiency evaluation covers memory footprint and inference-time overhead.The reported measures include total stored tokens, average tokens per unit, retrieval ratio, and additional prompt tokens per step.
  • Efficiency Metrics: Lower additional prompt tokens per step indicate reduced inference overhead and less reliance on large contextual inputs at comparable task performance.This metric captures the average increase in prompt tokens relative to a state-only prompt.

C.2. Evaluation on Function Calling Tasks

Skill-Pro is evaluated on BFCL v4 to test whether learned procedural Skills generalize to diverse function-calling scenarios. It achieves the highest reported accuracy among the compared reasoning baselines.

  • Benchmark and Setup: BFCL v4 evaluates single-turn and multi-turn execution, parallel tool calls, and irrelevant or uncallable queries.The benchmark tests precise tool invocation across diverse real-world applications.
  • Benchmark and Setup: Skill-Pro is compared against State, ReAct, and CoT reasoning baselines.These are the three reasoning baselines used in the main experiments.
  • Results: 0.433 accuracy: Skill-Pro outperforms ReAct (0.383) and CoT (0.367) on BFCL v4.Accuracy is reported as Mean±Std Dev in Table 5.
  • Results: The learned Skills capture reusable execution logic, including structured parameter validation and systematic constraint checking, that transfers beyond the training environment.The reported result supports generalization to practical software tool-use scenarios.

D.2. How Skills Guide Primitive Actions

Skill-Pro Skills guide primitive action generation by checking candidate actions against environmental rules and feedback before execution. The described mechanism revises invalid actions into valid prerequisites, while its broader value is evaluated over extended horizons.

  • Skill-Guided Action Selection: With SelfConsistencyCheck, the agent changes “take tomato 1 from fridge 1” to the valid prerequisite “open fridge 1” when the fridge is closed.The Skill detects the environmental constraint before object retrieval.
  • Scope of the Example: The qualitative ALFWorld example illustrates action-selection mechanics rather than efficiency gains at individual steps.The paper locates the value of procedural memory in reuse across recurring situations and extended-horizon task performance.
  • Skill Representation and Control: Skills are explicitly represented with initiation conditions, execution procedures, and termination conditions that govern when they activate and stop.The termination controller returns DONE when termination is met or initiation is no longer valid, and CONTINUE otherwise.
  • Skill-Guided Action Selection: A procedural Skill proposes an action, checks it against known rules or feedback, revises violations, and outputs the final consistent action.Execution terminates when an action passes all self-consistency checks.
  • Skill Evolution: Semantic gradients diagnose execution outcomes and map root causes to updates of initiation, policy, or termination components.Components remain unchanged when no update is required, and gradients provide concrete instructions.
  • Skill Evolution: Skill evolution aggregates recurring weaknesses across gradients, then refines existing logic or discovers a new Skill structure when the current one is irrelevant.The process updates initiation, policy, and termination while preserving the core intent during refinement.

F. Discussion and Limitations

The discussion identifies explicit, readable Skills as the current representation and describes future movement toward more implicit or directly executable procedures. PPO Gate compatibility depends on API access to output-token log-probabilities.

  • Skill Representation: Current Skills are explicit and readable rather than the largely implicit procedural memory described in human cognition.The explicit form supports learning, inspection, and system-level control.
  • Skill Representation: Future work may compress frequently reused Skills into executable code modules, parameterized procedures, or latent control policies.The paper also proposes decoupling mature Skill execution from language generation while retaining explicit representations for learning and debugging.
  • API Compatibility: Providers that do not expose log-probabilities remain incompatible with the current PPO Gate, motivating prompt-based verification alternatives.Suggested alternatives include LLM-as-judge scoring and self-consistency filtering.
Loading 2602.01869v3…