PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving Programmatic Skill Networks

Haochen Shi, Xingdi Yuan, Bang Liu

arXiv:2601.03509v2cs.AIcs.NE

TL;DR

Embodied agents must continually acquire, refine, and reuse skills, but existing approaches have limitations in continual improvement and credit assignment. PSN organizes executable symbolic skills into an evolving network with repair, update-gating, and refactoring mechanisms; experiments report faster acquisition, reduced forgetting, and stronger compositional generalization.

  • Problem

    Existing skill libraries often lack principled continual improvement, and agents lack unified mechanisms for credit assignment across compositions, program repair, and structural reorganization.

    Source: §1 · p. 1

  • Method

    PSN represents skills as executable symbolic programs in an evolving compositional network and combines trace-based fault localization, maturity-aware update gating, and canonical structural refactoring.

    Source: §1 · pp. 1–2, §2.4 · p. 4

  • Results

    Experiments on Minecraft and Crafter report faster skill acquisition, reduced forgetting, and superior compositional generalization.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The authors conclude that neural-network optimization principles may inform the design of symbolic learning systems.

    Source: §6 · p. 13

    The current implementation operates in an effectively batch-size-one online regime, limiting parallelism in skill execution and reflection-driven optimization.

    Source: Limitations · p. 19

Abstract

from arXiv · show

We study continual skill acquisition in open-ended embodied environments where an agent must construct, refine, and reuse an expanding library of executable skills. We introduce the Programmatic Skill Network (PSN), a framework in which skills are executable symbolic programs forming a compositional network that evolves through experience. PSN defines three core mechanisms instantiated via large language models: (1)~\opreflect for structured fault localization over skill compositions, (2)~progressive optimization with maturity-aware update gating that stabilizes reliable skills while maintaining plasticity for uncertain ones, and (3)~canonical structural refactoring under rollback validation that maintains network compactness. We further show that PSN's learning dynamics exhibit structural parallels to neural network training. Experiments on MineDojo and Crafter demonstrate robust skill reuse, rapid adaptation, and strong generalization across open-ended task distributions.

1 Introduction

The paper addresses the need for embodied agents to continually acquire, refine, and reuse skills, despite limitations in existing skill libraries and credit assignment. It introduces PSN’s executable skill network and three learning mechanisms, and frames their dynamics as structurally parallel to neural network training.

1 Introduction

  • Existing approaches often use flat libraries or static graphs without principled continual improvement, and lack unified ways to assign credit, repair programs, and reorganize hierarchical skills (Wang et al., 2024a; Yao et al., 2023).These limitations become relevant as new tasks arise.Source: §1 · p. 1
  • PSN represents skills as executable symbolic programs with explicit control flow, parameters, and preconditions, linked in a directed network that grows and reorganizes through learning.Its explicit computational graph supports trace-based credit assignment, maturity-aware stabilization, and structural refactoring.Source: §1 · pp. 1–2
  • PSN combines a network-aware planner, REFLECT fault localization over skill compositions, and network-level refactoring, with LLMs providing program synthesis capacity.The framework attributes continual-learning behavior to its architectural scaffolding rather than to the LLM itself.Source: §1 · p. 1
  • PSN’s fault localization, maturity gating, and structural refactoring have structural parallels to backpropagation, learning-rate scheduling, and neural architecture search, respectively.The authors suggest that neural-network optimization principles may also inform programmatic learning systems.Source: §1 · p. 2

2 Method

PSN continually constructs and updates a network of executable skills through planning, execution, failure-driven repair, and success-driven refactoring. Its mechanisms reuse reliable skills, localize repairs along execution traces, and validate structural changes before retaining them.

2 Method

  • The method treats tasks as a sequential stream of natural-language goals with varying difficulty, horizon, and compositional structure, and continually adapts reusable skills to solve them.Its online loop couples symbolic planning, execution, failure-driven repair, and success-driven refactoring.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Programmatic Skill Networks (PSN)

  • Each skill is a symbolic program with control flow, parameters, preconditions and postconditions, and invoked subskills; skills form a directed network of invocation links.Execution records feedback and invocation traces for subsequent learning.Source: §2.1 · p. 2

2.2 Network-Aware Hybrid Planner

  • The hybrid planner first reuses network skills through backward-chaining from the goal, expanding unmet preconditions and using skill reliability to break ties; it invokes an LLM-based forward planner when reuse cannot progress.Successful plans can be distilled into new skills.Source: §2.2 · p. 3

2.3 Execution and Trace Construction

  • The manager synthesizes candidate skills from plans using the task, current network, and execution history, then records execution traces with symbolic state snapshots and feedback.Observed success and failure states incrementally calibrate skill preconditions and postconditions.Source: §2.3 · pp. 3–4

2.4 Skill Optimization via Trace-Based Credit Assignment

  • On failure, REFLECT localizes faults along the executed invocation trace before applying symbolic edits to affected skills; a rolling buffer and maturity-aware gating constrain updates.Mature skills receive updates less often while immature skills remain plastic; the 0.6 value is a soft maturity pivot, with a 0.1 minimum update probability.Source: §2.4 · p. 4

2.5 Online Structural Refactoring

  • Online refactoring reorganizes existing programs through five canonical structural relationships, considering nearby and semantically related skills without introducing new behavioral logic.Tentative rewrites are checked on a sliding window of three recent tasks involving affected skills and reverted if success rate drops by more than 20%.Source: §2.5 · pp. 4–5

3 An Optimization Perspective on PSN

The paper interprets PSN’s continual-learning behavior as an implicit balance among task success, skill reliability, structural compactness, and semantic consistency. It maps its modules to neural-network optimization analogies while noting that the analogy is partial and applies to discrete symbolic programs.

3 An Optimization Perspective on PSN

  • PSN can be viewed as implicitly balancing task success, skill reliability, structural compactness, and semantic consistency, with modules locally improving different objective components.The objective is an interpretive lens rather than an explicitly optimized quantity.Source: §3 · p. 5
  • REFLECT assigns repair credit along executed skill paths, maturity gating reduces updates to reliable skills, and refactoring targets redundancy and missing abstractions.The paper relates these mechanisms to symbolic differentiation, adaptive learning rates, and symbolic architecture search.Source: §3 · pp. 5–6
  • PSN operates on fast behavioral-repair, intermediate skill-stabilization, and slower structural-refactoring timescales.The paper describes reliable skills as progressively stabilizing over 10–50 executions and refactoring as occurring every 5–10 successful executions.Source: §3 · p. 6
  • The neural-network analogy is partial: PSN uses discrete symbolic programs, structured edit proposals, and binary success/failure signals rather than continuous parameters, numeric derivatives, and differentiable losses.The authors present stability-plasticity tradeoffs, compositional credit assignment, and architecture search as principles that can also arise in structured representations.Source: §3 · p. 6

4 Experiments and Analysis

The experiments assess PSN on MineDojo and Crafter across task performance, continual learning, compositional generalization, and skill-network structure. Across these evaluations, PSN shows sustained task progress, skill retention, and compact skill reuse, including under different LLMs.

4.1 Experimental Setup

  • Evaluations span MineDojo and Crafter, comparing end-task performance, continual learning, compositional generalization, and network structure across LLM operators and agent baselines.Minecraft experiments also use Qwen3-Coder-Next alongside GPT-5-mini; the Crafter and Minecraft control APIs differ.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 Main Results

  • PSN progresses to higher-tier Minecraft tools with fewer attempts and lower variance, and achieves steadily increasing reward on Crafter.The supplied figures cover Minecraft tech-tree mastery and Crafter cumulative reward.Source: Figure 2 · p. 7, §4.2 · pp. 7–8, Figure 3 · p. 8

4.3 Generalization

  • After sequential task mastery, PSN retains previously learned skills, while Voyager’s retention rapidly degrades as new skills are acquired.Skill Retention Rate aggregates success on mastered tasks re-evaluated after subsequent tasks are mastered.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 Ablation Study

  • The optimizer supports reliable progress to later Minecraft objectives, while maturity gating stabilizes cumulative success by reducing updates to reliable skills and preserving plasticity in immature ones.Without the optimizer, performance is comparable to Voyager on early- and mid-stage tools but does not reliably reach diamond tools or obsidian.Source: §4.4 · p. 9, Figure 5 · p. 10
  • Canonical refactoring keeps PSN’s network compact; offline refactoring of Voyager reduces redundancy but yields a 0.6875 success rate versus 0.8462 for PSN with online refactoring.PSN applies semantics-preserving rewrites, while the comparison indicates that refactoring coupled to execution feedback is more effective than a one-time pass.Source: §4.4 · pp. 9–10

4.3 Generalization

  • PSN’s skill repertoire plateaus and later shrinks as it reuses existing skills and refactoring merges redundant helpers, unlike a variant that creates a new skill for every task.The comparison tests compositional generalization through backward chaining versus continual skill creation.Source: §4.3 · p. 9, Figure 6 · p. 11

4.5 REFLECT under LLM noise: a three-layer view

  • REFLECT’s noise defenses address different layers: a model-specific knowledge corrigendum, environment-validated diagnosis and repair, and a static API checker with contracts.The knowledge corrigendum contributes +33pp on Qwen3 versus +2pp on GPT-5-mini; the interface gate lowers bug rates from 65% to 35% in a 20-case replay.Source: §4.5 · pp. 10–12, Figure 7 · p. 11

4.6 Cross-model robustness and compositional reuse

  • Both GPT-5-mini and Qwen3 complete the tech tree under the same architecture, although Qwen3 requires deeper credit assignment and exhibits less compositional reuse.The weaker model propagates repair feedback through 5.0 skills on average versus 2.7 for the stronger model; reuse ratios asymptote near 0.15 ± 0.07 versus 0.4.Source: §4.6 · p. 12, Figure 8 · p. 12

5 Related Work

Prior work spans hierarchical reinforcement learning, programmatic agents, and neuro-symbolic or architecture-optimization methods. PSN combines executable symbolic skills with autonomous composition and optimization in a compositional network.

Skill Learning and Hierarchical RL

  • Unlike hierarchical RL approaches based on options, modular routing, or trajectory segmentation, PSN represents skills as executable programs with explicit control flow and preconditions.The passage identifies temporal abstraction and modular routing as established lines of work.Source: §5 · pp. 12–13

LLM-based Agents and Program Synthesis

  • Figure 9 contrasts PSN's reuse hubs with Voyager's flat retrieval library, where nearly every skill has fan-in 0.The comparison averages six runs per agent using Qwen3-Coder-Next.Source: Figure 9 · p. 13
  • PSN autonomously generates, composes, and optimizes skill code in a compositional network without hand-authored task-specific skills.Related approaches include learned causal mappings over predefined skills and LLM retrieval over hand-authored skills.Source: §5 · p. 13

Neuro-Symbolic Learning and Architecture Optimization

  • PSN embeds learning dynamics in symbolic programs and performs architecture-search-like refactoring under rollback validation, unlike differentiable neuro-symbolic systems.The related work also compares PSN's precondition-effect skill representation with OneLife's programmatic laws.Source: §5 · p. 13

6 Conclusion

PSN represents continually acquired skills as executable programs in an evolving compositional network. Its learning mechanisms parallel aspects of neural-network training, and experiments in Minecraft and Crafter report faster acquisition, reduced forgetting, and superior compositional generalization.

6 Conclusion

  • Experiments on Minecraft and Crafter demonstrated faster skill acquisition, reduced forgetting, and superior compositional generalization.The conclusion says these findings suggest neural-network optimization principles can inform symbolic learning systems.Source: §6 · p. 13
  • PSN's trace-based credit assignment, maturity-aware update gating, and canonical structural refactoring produce learning dynamics with structural parallels to neural-network training.The conclusion frames these as the framework's three mechanisms.Source: §6 · p. 13
  • PSN forms a compositional network of executable symbolic programs that evolves through experience.The stated conclusion is that neural-network optimization principles may inform symbolic learning-system design.Source: §6 · p. 13

Ethics Statement

The work evaluates autonomous skill acquisition in sandboxed Minecraft and Crafter simulators, with prompted LLMs as code-generation backends and no model fine-tuning or private-data training.

Ethics Statement

  • Experiments use sandboxed Minecraft and Crafter simulators, with no real-world deployment or direct safety risks.The authors note that autonomous code generation could produce unintended behaviors; PSN uses rollback validation and execution sandboxing as mitigations.Source: Ethics Statement · p. 14
  • The work uses prompted LLMs as code-generation backends and does not fine-tune models or train on private data.The experiments use publicly available game environments and APIs.Source: Ethics Statement · p. 14

Reproducibility Statement

The authors provide algorithm specifications, operator prompt templates, model identifiers, hyperparameters, evaluation task sequences, and representative optimization diffs; experiments use Minecraft 1.19.4 and Crafter.

Reproducibility Statement

  • Reproducibility materials include complete operator specifications, prompt templates, exact model identifiers, hyperparameters, task sequences, and representative optimization diffs.The two-phase optimization algorithm is specified in Appendix A, prompts in Appendix D, task sequences in Appendix H, and code diffs in Appendix G.Source: Reproducibility Statement · p. 14
  • Experiments are conducted on Minecraft 1.19.4 via MineDojo and Mineflayer, and on Crafter.Source: Reproducibility Statement · p. 14

Limitations

The appendix describes computational and theoretical limitations alongside a two-phase optimization procedure and canonical refactor patterns. Optimization separates feedback propagation from code updates, while refactoring consolidates skills through structural rewrites.

Limitations

  • Constrained resources limit PSN to effectively batch-size-one learning, restricting parallelism; formal guarantees for symbolic projection, convergence, and optimality remain unestablished.The authors expect increased compute and more efficient parallel execution to support larger batches and stronger guarantees in future iterations.Source: Limitations · p. 19

A.1 Feedback vs. Gradients

A.2 Phase I: Top-down Feedback Backpropagation

  • Phase I recursively follows execution-trace invocation links to propagate feedback and produce a pending optimization subgraph without modifying skill code.The subgraph pairs skills with their gradient proposals.Source: §A.2 · p. 19

A.3 Phase II: Bottom-up Gradient Application

  • Phase II applies proposed updates bottom-up, starting with leaf skills, and propagates optimization reports upward so parent skills can adapt to dependency changes.The context for an update aggregates reports from already-updated subskills.Source: §A.3 · p. 20

A.4 Algorithmic Interpretation; A.5 Discussion

  • Separating top-down credit assignment from dependency-consistent code modification is described as essential for stable optimization in deeply compositional, long-horizon tasks.The authors compare this separation to the backward–forward structure of neural backpropagation applied to symbolic skill networks.Source: §A.4 · p. 20, §A.5 · p. 20

B.1 Case A: Parametric Coverage; B.2 Case B: Behavioral / Subgraph Coverage

  • Parametric coverage replaces a specialized skill with a wrapper around a parameterized general skill, while behavioral coverage replaces duplicated composite logic with an existing skill call.Both rewrites reduce redundancy while preserving the specialized behavior or composite functionality described.Source: §B.1 · p. 20, §B.2 · pp. 21–22

B.3 Case C: Sibling Specializations; B.4 Case D: Common Subskill Extraction

  • Sibling specializations can be replaced by wrappers around a synthesized abstract skill, while repeated sub-operations can be extracted into a shared reusable skill.The patterns identify, respectively, a missing general operation and duplicated sub-operations across skills.Source: §B.3 · p. 22, §B.4 · pp. 22–23

B.5 Case E: Duplication Removal

  • Functionally equivalent skills are consolidated by retaining the higher-value implementation as canonical and redirecting links, with the redundant skill demoted to an alias or removed from planning.The equivalence may differ only in naming or minor surface variations.Source: §B.5 · p. 23

E Inline Knowledge Scaffolding Ablation

This appendix examines the role of injected knowledge in REFLECT’s diagnostics and illustrates PSN’s execution-trace-driven repairs across skill hierarchies. It also presents task sequences for temporal generalization and an identical-sequence comparison of offline and online refactoring.

E Inline Knowledge Scaffolding Ablation

  • Removing the tool-tier corrigendum reduced Qwen3-Coder-Next tier-mismatch diagnosis from 100% to 67% (+33pp), but GPT-5-mini scored 98% without it, a +2pp contribution.The Qwen3 comparison uses a cleaned baseline; sibling hints had inflated no-corrigendum accuracy to 83%, while pruning them left GPT-5-mini near-100% either way.Source: E Inline Knowledge Scaffolding Ablation · pp. 26–28
  • On the strong-LLM evaluation, removing inline scaffolding leaves REFLECT’s diagnostic correctness nearly unchanged, supporting an architectural rather than prompt-trick origin for its capability.The Qwen3 corrigendum instead patches documented Minecraft pretraining gaps, separating weaker-model scaffolding from the architectural mechanism.Source: E Inline Knowledge Scaffolding Ablation · pp. 27–28

F Additional Optimization Examples

  • PSN’s trace-based symbolic credit assignment supports localized fixes and coordinated optimization across skill hierarchies; the appendix presents recurring patterns rather than an exhaustive catalog.The taxonomy summarizes common failure signals and repair strategies.Source: F Additional Optimization Examples · p. 28, §F.1 · p. 28

F.2 Representative Optimization Cases

  • Representative repairs corrected craftWoodenPickaxe resource estimates, made ensureFlint fail fast, constrained chest withdrawals by inventory capacity, and validated crafting-station preconditions for ensureMetalIngots.The repairs respectively account for planks used in stick crafting, surface failures to upstream skills, adapt withdrawals to available space, and locate or construct missing stations.Source: §F.2 · pp. 28–29

F.3 Advanced Optimization: Cross-Skill Credit Assignment

  • In ensureRawIronAndFuel → ensureFuel, PSN co-optimizes parent and child skills: the parent verifies fuel postconditions and recovers when needed, while the child refines fallbacks and fuel handling.This coordinated repair localizes responsibility across skill boundaries and restores reliable delivery of the required resources.Source: §F.3 · p. 29, §G.5 · p. 40

G Detailed Code Diffs for Optimization Examples

  • The detailed diffs show fixes that include recalculating planks for stick crafting, removing ensureFlint’s unsafe manual fallback, limiting chest withdrawals to capacity, and ensuring a crafting table before furnace crafting.The appendix also provides tables summarizing optimization cases and mapping gradient signals to implemented fixes.Source: G Detailed Code Diffs for Optimization… · p. 31, Table 6 · p. 31, §G.1 · p. 32

G.5 Example 5: Cross-Skill Co-Optimization

  • PSN assigns credit to both hierarchy levels and optimizes them simultaneously to address insufficient fuel that causes downstream smelting failures.The example describes localizing responsibility across skill boundaries in a compositional hierarchy.Source: §G.5 · p. 40

H Task Sequences

  • The temporal curriculum progresses from mining wood through crafting an iron pickaxe, while refactor evaluation compares offline and online methods on an identical sequence without retraining.The evaluation sequence adds crafting planks and mining cobblestone before making a stone pickaxe.Source: H Task Sequences · p. 40

I Comparison with ODYSSEY

PSN learns skills autonomously from scratch and reaches the diamond tool in fewer iterations than ADAM and Voyager* in the reported comparison. On the Minecraft tech tree, PSN’s success rate rises with attempts, and its trace-based optimization involves coordinated repairs across multiple skills.

I Comparison with ODYSSEY

  • 35±16 iterations (6/6 runs) let PSN reach diamond tool, fewer than ADAM (74±23) and Voyager* (99±36, 2/6); PSN uses 30 autonomously learned skills.The comparison contrasts PSN’s autonomous skill generation with the hand-authored task-specific skills used by ADAM and ODYSSEY.Source: Table 8 · p. 41, I Comparison with ODYSSEY · p. 41
  • PSN starts with lower SR@1 but reaches 100% at SR@3 through trace-based optimization, matching ODYSSEY’s final reliability with 30 rather than 183 skills.The authors report this as a 6× reduction in skill count.Source: Table 9 · p. 41, I Comparison with ODYSSEY · p. 41

J.1 Per-Episode Optimization Scope

  • 56.9% (GPT-5-mini) and 80.8% (Qwen3-Coder-Next) of optimization episodes involve coordinated repairs across multiple skills; multi-skill episodes average 2.7 and 5.0 skills, respectively.The table caption identifies deeper propagation for Qwen3-Coder-Next than GPT-5-mini.Source: §J.1 · p. 41, Table 10 · p. 42

J.2 Error Mode Taxonomy

The appendices analyze REFLECT’s error patterns and structural credit assignment, then report refactoring safety, cost, agent-comparison scope, and the learning-dynamics objective. Together, these analyses characterize how errors are repaired, network changes are constrained, and network properties are measured.

J.2 Error Mode Taxonomy

  • Across 1,059 optimization records, resource miscalculation led (49.9%), while cross-skill contract errors made up ∼17% and triggered multi-skill repair episodes.LLM hallucination was extremely rare (0.3%, 3 cases across all runs).Source: §J.2 · p. 42, Table 11 · p. 42

J.3 Structural Credit Assignment

  • Multi-skill repair episodes occurred in 56.9% of GPT-5-mini episodes and 80.8% of Qwen3-Coder-Next episodes, consistent with recursive trace-based repair across the skill graph.REFLECT analyzes invocation traces top-down and optimizes identified skills bottom-up.Source: §J.3 · p. 42

K Rollback and Safety Statistics

  • Pre-application checks rejected 31% of proposals; post-application rollbacks were rare at 3.1% and 6.7% of iterations, with the weaker model experiencing approximately double the rollback rate.Post-application monitoring reverts changes when success drops by more than 20% over a sliding window of three recent tasks.Source: K Rollback and Safety Statistics · p. 43, Table 12 · p. 43

L Refactoring Case Distribution

  • Behavioral coverage and common-subskill extraction accounted for 43% and 35% of refactoring cases, while duplication removal was 1%; validation rejected 31% of proposals.Of 139 proposed refactors, 96 passed pre-application validation.Source: L Refactoring Case Distribution · p. 43, Table 13 · p. 43

N Minecraft Agent Paradigm Taxonomy

  • Controlled comparison with PSN requires Mineflayer, code generation, compatible action spaces, and comparable metrics; browser and text-domain methods differ in environment and task structure.The taxonomy identifies only code-generating, Mineflayer-based agents as supporting controlled comparison.Source: Table 15 · p. 44, N Minecraft Agent Paradigm Taxonomy · pp. 44–45

O Learning Dynamics Decomposition

  • The section tracks four components of the composite objective across training iterations to examine the network’s learning trajectories.The objective is presented as a diagnostic of network health, not an explicitly optimized quantity.Source: O Learning Dynamics Decomposition · p. 45, §O.1 · p. 45

O.1 Component Definitions

  • The objective weights task success, skill reliability, compositional reuse, and refactoring consistency at 0.4, 0.3, 0.2, and 0.1, respectively.The weights were hand-selected for interpretability; the objective combines rolling task success, mean skill reliability, shared-skill ratio, and refactoring success rate.Source: §O.1 · p. 45

O.2 Empirical Trajectories

PSN’s composite objective remains stable as training tasks become more complex. The reported trajectories show task success and growing compositional reuse alongside refactoring consistency, while skill reliability accumulates more slowly.

GPT-5-mini: stable learning under increasing complexity

  • J(N) stays near 0.5 as tasks escalate from resource gathering to deeply compositional objectives and the skill graph grows from 1 to ∼40 nodes.The graph reaches depth up to 6 during this progression.Source: §O.2 · p. 45, Figure 17 · p. 46
  • Rtask remains near 1.0, Rstruct grows from 0 to ∼0.4, and Rcons stabilizes around 0.7 as task complexity increases.On Qwen3, Rstruct plateaus at 0.15 ± 0.07; Rreliab stays near zero because reliability accumulates more slowly than task success.Source: §O.2 · pp. 45–46
  • The stable objective is described as a compensatory dynamic: increasing compositional reuse accompanies escalating task difficulty while task performance remains high.The paper compares this pattern to a well-regularized neural network learning increasingly complex patterns.Source: §O.2 · p. 46

Weight sensitivity

  • J(N) remains within a narrow range under all tested alternative weightings rather than degrading as task complexity increases.The paper reports testing four weighting schemes.Source: §O.2 · p. 46, Table 16 · p. 47
Loading 2601.03509v2…