PaperlayerView exact revision on arXiv

Source-linked AI summary

Rep2Skill: Representation-Guided Skill Self-Evolution for LLM Agents

Kaixing Zhang, Changming Li, Yingdong Shi, Zheng Zhang, Kaitao Song, Wenjie Shi, Jingang Wang, Kan Ren

arXiv:2609.39149v1cs.AI

TL;DR

Existing skill evolution relies on textual trajectories and task outcomes, leaving internal representations outside the optimization loop while asking optimizers to diagnose failures and revise skills. Rep2Skill models representation trajectories and turns their signals, alongside execution context, into feedback for targeted revisions. It consistently outperforms text-only approaches across two benchmarks in self-evolution experiments.

  • Problem

    Text-based skill evolution relies on trajectories and task outcomes, leaving internal representations outside the optimization loop and requiring the optimizer to diagnose failures and propose revisions.

    Source: §1 · pp. 1–2

  • Method

    Rep2Skill compares representation trajectories across grouped rollouts to identify failure-relevant turns and verbalizes signals with trajectory context as textual revision guidance.

    Source: §1 · p. 2

  • Results

    Rep2Skill consistently achieves the best downstream performance across settings, and representation guidance improves failure-turn attribution, reaching 0.838 AUROC and 0.876 AUPRC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support representation space as complementary evidence for targeted skill refinement beyond text-only reflection.

    Source: §5 · p. 8

    The method requires access to model internal representations, experiments focus primarily on Qwen models, and representation modeling adds about 50 rollouts plus lightweight training.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Textual skills enable large language model (LLM) based agents to accumulate reusable procedural knowledge without updating model parameters. Yet existing skill evolution remains largely confined to the text space: an optimizer must diagnose success and failure patterns, and revise skills solely from long execution trajectories and sparse task outcomes. This text-only paradigm leaves the agent's internal representations, which contain rich records of its evolving execution state, outside the skill optimization loop. We ask whether an agent can improve its external textual skills by reflecting on its own internal representations. We introduce Rep2Skill, a representation-guided framework for self-evolution on agent skills. Specifically, upon the collected agent rollouts, Rep2Skill models their internal model representation trajectories to localize turns that deviate from successful execution dynamics, and it further interprets these signals alongside the execution contexts as actionable textual feedback for targeted skill revision. Experiments on two agent environments with two open-source LLMs show that Rep2Skill consistently outperforms text-only approaches in the self-evolution setting, where the same LLM serves as both executor and optimizer without a stronger external model. This establishes a promising direction moving agent self-improvement beyond text-only reflection.

1 Introduction

Rep2Skill adds internal representation trajectories to textual skill evolution, using grouped rollouts to guide targeted revisions. Across two agent benchmarks and two open-source LLMs, it consistently outperforms text-only optimization.

1 Introduction

  • Existing skill-evolution methods rely on trajectories and task outcomes, requiring optimizers to identify failure causes and determine appropriate skill revisions.This diagnostic burden is especially substantial when the optimizer has limited capability.Source: §1 · pp. 1–2
  • Multiple rollouts for the same task provide relative evidence from representation-dynamics differences to localize failure-relevant turns and target skill revisions.This treats representations as complementary evidence beyond observable execution trajectories.Source: §1 · p. 2
  • Rep2Skill models representation dynamics, compares potentially failure-relevant turns across rollouts of the same task, and turns signals plus trajectory context into revision guidance.The resulting textual feedback guides skill evolution for subsequent executions.Source: §1 · p. 2
  • Rep2Skill consistently achieves the best downstream performance across evaluated settings, while representation guidance improves failure-turn attribution over text-only attribution.The analyzer reaches 0.838 AUROC and 0.876 AUPRC for turn-level attribution.Source: §1 · p. 2
  • The paper frames representation space as a complementary source for skill optimization and reports that gains are particularly large with a limited-capability skill optimizer.It evaluates Rep2Skill on ALFWorld and WebShop with two Qwen models.Source: §1 · p. 2

2 Related Work

Prior skill-evolution methods convert execution experience into reusable textual skills, but primarily rely on observable trajectories, outcomes, and validation scores. Rep2Skill adds internal representations as complementary feedback, drawing on work that identifies information in model activations and translates it into language.

2 Related Work

  • Experience-driven skill methods refine procedural knowledge from traces and outcomes, but derive learning signals from observable artifacts, leaving internal representations underused.EvoSkill attributes failed rollouts to missing capabilities, Trace2Skill aggregates lessons across executions [Ni et al., 2026], and SkillOpt updates textual states with validation [Yang et al., 2026a].Source: §2 · pp. 2–3
  • Rep2Skill verbalizes representation signals into textual feedback, complementing execution artifacts for fine-grained skill evolution.This draws on latent-to-language approaches such as Activation Oracles and Natural Language Autoencoders.Source: §2 · p. 3
  • Prior work finds that model representations encode semantic, behavioral, and agent-state information, sometimes before it appears in observable trajectories.Examples include signals about tool-use decisions, execution failures, and task-relevant states.Source: §2 · p. 3

3 Method

REP2SKILL extends textual skill self-evolution with internal representation trajectories, using grouped rollouts to identify turns for feedback-guided, validation-gated skill revisions.

3.1 Preliminaries: Textual Skill Self-Evolution

  • In the self-evolution setting, the same frozen LLM executes tasks and optimizes textual skills without a stronger external model, while REP2SKILL also uses its execution representations.Standard skill evolution relies primarily on observable trajectories and task-level outcomes.Source: §3.1 · p. 3

3.2 Attribution in Representation Space

  • For each task and skill, grouped rollouts provide behavioral contrasts; a model trained on successful executions scores projected hidden-state prediction errors to select high-deviation turns.These turns serve as localized credit signals for skill revision.Source: §3.2 · p. 4

3.3 Representation-to-Text Verbalization

  • Representation-based attribution yields more accurate fine-grained credit assignment than trajectory-text-only attribution, while analyzer feedback grounds high-scoring turns in trajectory and skill evidence.The analyzer identifies potential revisions only when supported and can conclude that behavior is already covered or evidence is insufficient.Source: §3.3 · p. 5

3.4 Skill Evolution in Text Space

  • The optimizer consolidates representation-guided feedback into candidate skills and accepts the best-performing candidate only when it improves on the current skill.Otherwise, the current skill is retained.Source: §3.4 · p. 5

4 Experiment

Rep2Skill is evaluated as representation-guided skill evolution across ALFWorld and WebShop, with analyses of turn attribution and design choices. It achieves the highest mean performance across both benchmarks and models, while further analyses examine attribution accuracy, rollout diversity, ablations, and a targeted revision case.

4.1 Experimental Setup

  • Evaluation uses ALFWorld and WebShop, compares No Skill with three skill-evolution baselines, and tests two Qwen models in self-evolution runs.Results are averaged over three independent runs; each query uses four rollouts for optimization.Source: §4.1 · p. 5

4.2 Representation-Guided Skill Evolution (RQ1)

  • Rep2Skill has the highest mean performance on both benchmarks with both models, including ALFWorld success rates of 52.24% (Qwen3-4B) and 69.90% (Qwen3.5-9B).On WebShop, it achieves success rates of 16.33% and 23.33%, and task scores of 49.79 and 53.56, with Qwen3-4B and Qwen3.5-9B, respectively.Source: Table 1 · p. 6, §4.2 · p. 6
  • On WebShop, some text-only methods underperform No Skill, whereas Rep2Skill improves both success rate and task score relative to No Skill.The text identifies SkillOpt as underperforming on task score with Qwen3-4B and Trace2Skill and EvoSkill as underperforming on both metrics with Qwen3.5-9B.Source: §4.2 · p. 6

4.3 Why Does Representation Help Skill Evolution? (RQ2)

  • 0.838 AUROC, 0.876 AUPRC, and 88.22% Precision@Top-3: representation-derived guidance improves the same analyzer over trajectory text alone.Representation scores alone improve AUROC to 0.592.Source: Table 2 · p. 7, §4.3 · p. 7
  • Text + Rep. Guidance achieves the highest precision and recall across selection budgets K ∈ {1, 3, 5, 7, 10}.Precision@K and Recall@K are macro-averaged over eligible trajectories.Source: Figure 3 · p. 7

4.4 Further Analyses of Rep2Skill (RQ3)

  • Within-query action disagreement reaches 82.7% at turn 5, and four rollouts of one ALFWorld task follow distinct paths in representation space.The analysis covers 84 groups of Qwen3-4B rollouts across 12 ALFWorld tasks under an empty skill.Source: Figure 4 · p. 7, §4.4 · p. 7
  • On ALFWorld, the full method performs best; removing verbalization, representation attribution, or group-wise rollout evidence reduces performance.The ablations identify group-wise comparison, representation-space attribution, and semantic interpretation of attributed evidence as complementary ingredients.Source: Table 3 · p. 7, §4.4 · pp. 7–8
  • Representation attribution localizes repeated cabinet searches, enabling a targeted rule to check countertops, drawers, or the sink; the agent finds the mug in 8 steps.The text-only revision instead encourages more thorough search and repeats the failure after 30 steps.Source: Figure 5 · p. 8, §4.4 · p. 8

5 Conclusion

REP2SKILL brings internal representation trajectories into textual skill evolution, using them to guide revision. On ALFWorld and WebShop with two open-source LLMs, representation guidance leads to more effective skill evolution than text-only feedback.

5 Conclusion

  • REP2SKILL outperforms text-only feedback for skill evolution on ALFWorld and WebShop with two open-source LLMs.Turn-level analysis and component ablations further support the roles of representation guidance in skill self-evolution.Source: §5 · p. 8
  • The framework grounds internal representation signals in deviations from successful execution dynamics to guide downstream skill revision.It couples representation-space analysis with text-space evolution.Source: §5 · p. 8
  • Turn-level analysis and component ablations support the respective roles of representation guidance in skill self-evolution.Source: §5 · p. 8

Limitations

The method requires access to model internal representations, and the experiments primarily use Qwen-family models. The appendix also identifies Algorithm 1 as a summary of the method’s main steps.

Limitations

  • Direct applicability may be limited to white-box models because the method requires access to internal representations.An accessible proxy model is suggested as a feasible alternative for black-box models.Source: Limitations · p. 9
  • Experiments primarily use Qwen-family models, so evaluation across more architectures would help establish broader generality.Source: Limitations · p. 9

A Algorithms of REP2SKILL

B.1 Details on Representation Attribution

REP2SKILL models successful representation trajectories and scores turns that deviate from their learned dynamics. It extracts turn-level hidden states, projects them into a latent space, and uses prediction errors to select critical turns for feedback.

B.1 Details on Representation Attribution

  • The method extracts layer-24 hidden states by averaging tokens within each turn, using only the trajectory body and excluding the skill, reward, and failure information.The frozen agent processes the trajectory with an additional causal forward pass.Source: §B.1 · p. 11
  • PCA and per-dimension standardization map high-dimensional turn representations to 64-dimensional latent states, fitted only on successful training turns.The projection and trajectory model are fitted separately for each agent-model and environment combination.Source: §B.1 · p. 11
  • A Neural CDE models the latent trajectory using a control path constructed from time-augmented observations and cubic interpolation.The control path through turn t depends only on the first t turns, supporting prefix-based prediction.Source: §B.1 · p. 11
  • The trajectory model is trained on successful rollouts, so large prediction errors indicate deviations from learned successful-execution dynamics.Training trajectories are collected offline from disjoint tasks, with around 50 successful trajectories per dataset.Source: §B.1 · p. 11
  • For each rollout, the three highest-scoring turns from turn two onward form the critical-turn set; short rollouts without enough scored turns receive no representation feedback.Source: §B.1 · p. 11

B.2 Details on Skill Updates

Skill updates use group-wise rollout feedback, consolidation, bounded edit selection, and validation, with the same frozen LLM performing the stages. A representation analyzer turns selected trajectory evidence into structured diagnoses and rejects feedback that fails grounding checks.

B.2 Details on Skill Updates

  • Skill evolution follows group-wise reflection, consolidation, edit selection, application, and validation, with the same frozen LLM used throughout.The process uses the initial skill and sampled rollout groups, and disables SkillOpt’s slow-update and meta-skill modules.Source: §B.2 · pp. 11–12
  • Patches consolidate failure-driven and success-driven edits separately, resolving duplicates and conflicts and prioritizing edits supported across groups.Source: §B.2 · pp. 11–12
  • When a consolidated patch exceeds four edits, the optimizer retains the top four by systematic impact, complementarity, generality, and actionability.The fixed edit budget acts as a textual learning rate.Source: §B.2 · p. 12
  • A candidate skill replaces the current one only when its selection score is strictly higher; the best-scoring skill is reported on the test split.A step buffer also records prior failure patterns, rejected edits, and score changes for subsequent reflection.Source: §B.2 · p. 12

B.3 Details on Representation Analyzer

  • The representation analyzer, the same frozen LLM as the agent, receives the current skill and rollout evidence and returns a summary plus at most three diagnoses.Diagnoses record decision context, relevant skill rules, trajectory evidence, counterevidence, conditions, a modification hypothesis, and a recommendation.Source: §B.3 · pp. 12–13
  • Programmatic checks require exact skill and trajectory quotes and valid step references; ungrounded responses are discarded, and execution-deviation or insufficient-evidence diagnoses cannot recommend changes.The analyzer distinguishes suspect rules, rule conflicts, execution deviations, missing rules, and insufficient evidence, and is instructed not to force a skill change.Source: §B.3 · pp. 13–14

C Experiment Setup Details

The experiments use ALFWorld and WebShop with fixed train, selection, and test splits. Each run evolves skills over three epochs using four rollouts per task and repeats the run with three seeds.

Benchmarks

  • ALFWorld uses 39 training tasks, 18 selection tasks, and 134 valid_unseen test tasks; WebShop uses 50 training, 20 selection, and 100 test instructions.The ALFWorld tasks are stratified across six task families; WebShop uses the first 100 instructions of the official test split.Source: C Experiment Setup Details · p. 14

Evaluation

  • WebShop reports average task score scaled to [0, 100] and success rate, with all splits frozen across methods, models, and seeds.An episode counts as successful only if it receives the full reward.Source: C Experiment Setup Details · p. 14

Evolution protocol

  • Each run evolves the skill for three epochs, sampling four rollouts per task and making one validation-gated update per iteration.ALFWorld covers its training set once per epoch; WebShop uses batches of five tasks and ten iterations per epoch.Source: C Experiment Setup Details · p. 14

Model configuration

  • Runs use seeds 42, 43, and 44, and the same model serves as actor, representation analyzer, and skill optimizer.Models run locally in bfloat16 with thinking mode disabled; representation extraction uses a separate Hugging Face process and dedicated GPU.Source: C Experiment Setup Details · p. 14

D.1 Diversity Analysis

Rollouts for the same task and skill diverge in both action sequences and representation space. The analysis uses this within-group variation to motivate relative comparisons when locating deviations from successful execution dynamics.

Diversity Analysis

  • The analysis covers 84 groups of four Qwen3-4B rollouts from 12 ALFWorld training tasks, using layer-24 hidden states and cosine distance.Text-space comparisons include normalized action matches, action-sequence edit distance, and TF-IDF cosine distance over turn contents.Source: §D.1 · p. 14
  • By turn 10, 98.5% of rollouts have unique action prefixes, while mean pairwise representation cosine distance grows from 0.040 at turn 1 to about 0.15 at turns 5–10.The distance is the mean pairwise cosine distance of hidden states.Source: §D.1 · p. 14, Table 5 · p. 15
  • Across the first 15 turns, median within-group action edit distance is 0.607 and median cumulative-prefix TF-IDF distance is 0.387.Action edit distance treats each action as one sequence element.Source: §D.1 · p. 14, Table 5 · p. 15
  • 35.3% of action pairs disagree at turn 1, rising to 82.7% by turn 5, when 88.4% of rollouts have unique action prefixes.Table 5 summarizes within-group action disagreement and unique-prefix rates for Qwen3-4B rollouts on ALFWorld.Source: §D.1 · p. 14, Table 5 · p. 15
  • In six illustrative task-family groups, rollouts spread into different representation regions within the first few turns, and successful and failed rollouts often follow different paths.Even all-failure groups show different individual paths, suggesting that the rollouts fail in different ways.Source: §D.1 · p. 15, Figure 6 · p. 16
Loading 2609.39149v1…