PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation

Wenhao Wang, Yanyan Li, Jiawei Yuan

arXiv:2610.10812v1cs.RO

TL;DR

Enumerating representative task-solution pairs makes it difficult for SLMs to generalize to diverse, unseen robot tasks. Skill-SLM decomposes instructions into subtasks, selects and composes skills, and trains SLM agents for these stages. It outperforms baselines on UAV tasks, especially unseen tasks, and also demonstrates generality on ground vehicles.

  • Problem

    Enumerating task-solution examples makes dataset construction difficult and limits generalization to diverse tasks, including unseen capability combinations and underrepresented tasks.

    Source: §I · p. 1

  • Method

    Skill-SLM builds a skill library with an operational-skill-aware CFG, trains decomposition and orchestration SLM agents using LLM-generated datasets, and progressively generates code while preserving context and dependencies.

    Source: §I · pp. 1–2

  • Results

    Skill-SLM achieves the highest average task success rate across all three UAV task sets, outperforms baselines particularly on unseen tasks, and also performs across ground-vehicle tasks.

    Source: §I · p. 2, 1) Overall Result · p. 6

  • Takeaways & Limitations

    The results demonstrate a skill-based approach that generalizes across UAV and ground-vehicle operations, particularly when tasks require recombining learned capabilities.

    Source: §V · p. 8

Abstract

from arXiv · show

Small language models (SLMs) have been increasingly adopted for onboard robot operation because they enable intelligent decision-making. However, existing approaches are mainly distillation-oriented and rely on enumerating representative task-solution pairs. This makes dataset construction difficult and limits generalization to diverse robot tasks whose possible forms grow rapidly. This paper proposes Skill-SLM, a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem. Given a natural language task instruction, Skill-SLM decomposes the task into subtasks, selects appropriate skills from the skill library, and orchestrates the selected skills into executable robot operations. First, to support the skill-driven workflow, we propose a novel robot operational skill aware context-free grammar (CFG) to extract the skills required to accomplish tasks and build the skill library accordingly. Then, we configure LLM teachers to induce and synthesize training datasets for the SLMs, enabling SLMs to decompose tasks and orchestrate skills reliably. Additionally, we employ a progressive skill orchestration strategy to improve the reliability of skill implementation and overall robot operation. Experiments on UAV operation tasks indicate that Skill-SLM substantially outperforms distillation-oriented baselines, especially on unseen tasks that require generalization of capabilities. Additional experiments on ground vehicle tasks further demonstrate that Skill-SLM can be applied to different robot platforms.

I. INTRODUCTION

Enumerated task-solution examples make it difficult for SLMs to generalize across diverse, unseen robot tasks. Skill-SLM instead decomposes tasks and composes operational skills, with experiments reporting stronger performance on UAV and ground-vehicle tasks.

I. INTRODUCTION

  • Enumerating task-solution examples limits generalization to unseen capability combinations and underrepresented tasks, while continually adding examples is costly and difficult to anticipate.The paper also notes that SLM memorization becomes unreliable as learned knowledge grows.Source: §I · p. 1
  • SLMs can misinterpret task-relevant context, select incorrect skills, overlook skill dependencies, and fail to coordinate skills into coherent execution.Earlier skills may establish targets, variables, or robot states that later skills must preserve.Source: §I · p. 1
  • Skill-SLM uses a decomposition agent to divide instructions into subtasks and select skills, then an orchestration agent to turn those selections into executable robot operations.Unlike end-to-end instruction-to-operation mapping, the framework represents tasks compositionally.Source: §I · pp. 1–2
  • An operational-skill-aware CFG supports the skill library, LLM-generated datasets train the two SLM agents, and progressive orchestration preserves context, intermediate variables, robot states, and cross-skill dependencies.The decomposition dataset pairs instructions with subtasks and selected skills; the orchestration dataset pairs subtask descriptions and skill specifications with robot-operation code.Source: §I · p. 2
  • Skill-SLM consistently achieves higher task success rates than baselines, particularly on unseen tasks requiring knowledge recombination, and also generalizes to ground-vehicle operations.Qwen3-8B reaches 100% on Seen-Pattern and New-Combination UAV tasks and 94% on New-Capability tasks; ground-vehicle success rates are 100%, 95%, and 100%, respectively.Source: §I · p. 2

II. RELATED WORK

Prior work studies reusable agent skills, skill discovery, and SLM-enabled robotics. Skill-SLM builds on these areas by using reusable capabilities instead of relying on memorized task-solution mappings.

A. Agent Skills

  • Agent-skill approaches package reusable knowledge for agents to load when relevant, while earlier skill methods often focus on learned skills in constrained environments.SkiROS2 represents parameterized procedures with explicit execution conditions; LRLL learns skills through exploration and summarization.Source: A. Agent Skills · p. 2

B. Skill Discovery and Characterization

  • Prior skill-taxonomy methods largely use manual categories, whereas NavNuances uses an LLM-refined, manually designed CFG; Skill-SLM instead infers operations from a task corpus to construct its CFG.The cited approaches include task taxonomies and analyses of recurring task-skill structures and household activities.Source: B. Skill Discovery and Characterization · p. 2

C. SLM-enabled Robotics

  • Existing SLM robotics methods transfer LLM capabilities but largely train models to memorize enumerated task-solution mappings; Skill-SLM instead recombines reusable capabilities.The related work describes instruction-to-code training and SLM planning, as well as parameter-efficient training for a planner.Source: C. SLM-enabled Robotics · pp. 2–3

III. METHOD

Skill-SLM turns a task instruction into a subtask-and-skill plan, then progressively generates code for its selected skills. Its skill-discovery process induces and verifies a CFG from a task corpus.

A. Overview

  • The decomposition agent produces a plan pairing each subtask with a selected library skill, and the orchestration agent progressively generates code segments that are concatenated into final robot-operation code.The workflow begins with a task instruction and aims to produce code for the robot to complete it.Source: A. Overview · p. 3

1) Operational Skill Discovery:

  • An LLM induces a CFG from task instructions and verifies that it represents all tasks while identifying semantically distinct skills without unnecessary redundancy.The CFG is iteratively revised after verification, and the process outputs the CFG and skill list.Source: 1) Operational Skill Discovery · pp. 3–4

2) Skill Construction:

Skill documents ground discovered skills in robot APIs and specify how to use them. Progressive orchestration generates code subtask by subtask while retaining context and dependencies.

2) Skill Construction:

  • Each skill is grounded in robot APIs by a human expert and documented with its functionality and dependencies.The document separates compact metadata from a procedural body containing purpose, selection conditions, inputs, preconditions, implementation procedure, expected outputs, and operational constraints.Source: 2) Skill Construction · p. 4

C. Progressive Skill Orchestration

  • Progressive orchestration generates code for each planned subtask conditioned on the global task, local subtask, selected skill procedure, robot actions, and code from preceding subtasks.Retaining earlier code lets the SLM reference variables, targets, and intermediate results, helping preserve cross-skill dependencies and improve composition coherence and reliability.Source: C. Progressive Skill Orchestration · p. 4

D. Dataset Synthesis and SLM Training

Skill-SLM uses LLM teachers to synthesize decomposition and orchestration training data, with a refinement stage to improve data correctness. Its decomposition dataset pairs task instructions with plans and includes skill metadata as selection context.

D. Dataset Synthesis and SLM Training

  • LLM teachers synthesize decomposition and orchestration datasets, followed by a refinement stage that evaluates and refines the generated data.The refinement stage is introduced to ensure data correctness.Source: D. Dataset Synthesis and SLM Training · p. 4
  • Dataset construction first induces task instructions with the CFG, then has LLMs synthesize examples of task decomposition and skill orchestration.Source: D. Dataset Synthesis and SLM Training · p. 4

1) Decomposition Dataset Synthesis:

  • For decomposition training, an LLM uses each task instruction and available-skill metadata to generate a concise subtask plan with an appropriate skill selected for each subtask.The dataset includes metadata for all available skills as context during skill selection.Source: 1) Decomposition Dataset Synthesis · p. 4

2) Orchestration Dataset Synthesis:

The orchestration dataset trains the SLM to generate executable code one plan step at a time, using the task, skill procedure, APIs, and previously generated code. Skill-SLM trains separate LoRA adapters for decomposition and orchestration and evaluates onboard-capable SLMs on two robot platforms.

2) Orchestration Dataset Synthesis:

3) SLM Training:

  • Two separate LoRA adapters train the decomposition SLM to generate plans and the orchestration SLM to turn plans into executable robot-operation code.Source: 3) SLM Training · p. 5

A. Experimental Setup

  • Experiments deploy 4-bit-quantized SLMs on Jetson Orin NX hardware aboard a Holybro X500 V2 UAV and a ROSMASTER X3 ground vehicle.The models evaluated are Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B, and Qwen3-8B.Source: A. Experimental Setup · p. 5, Figure 4 · p. 5
  • An LLM induces a CFG from 50 task instructions to identify operational skills, which are documented in a skill library.Source: A. Experimental Setup · p. 5

2) Training Set:

The training set is synthesized through LLM-generated task instructions, task-to-skill decomposition, and subtask-to-code orchestration. Evaluation separates familiar patterns, recombinations of learned knowledge, and tasks requiring newly introduced skills; Ro-SLM and PRISM provide comparison baselines.

2) Training Set:

  • The training data contains 300 task-subtask-skill pairs for decomposition and 963 subtask-code pairs for orchestration.An LLM configured with the CFG induces the 300 task instructions, then LLM teachers generate the pairs.Source: 2) Training Set · p. 5

3) Test Task Sets:

  • New-Combination tasks test recombination of learned knowledge, while Seen-Pattern tasks have at least three in-distribution examples per test task.Source: 3) Test Task Sets · p. 5
  • New-Capability tasks require skills absent from distillation and test whether skill-library augmentation supports new tasks without additional distillation.Source: 3) Test Task Sets · p. 5
  • Ro-SLM generates complete robot-operation code end-to-end, whereas PRISM distills an LLM planner into an SLM and uses a compiler to produce executable code.Both baselines use the same 300 training task instructions as Skill-SLM.Source: 3) Test Task Sets · p. 5

4) Baselines:

The evaluation compares robot operations against ground-truth state transitions and reports operation-level progress and full task completion. For fair New-Capability comparisons, baselines receive knowledge of new skills through their prompts or compiler.

4) Baselines:

  • For New-Capability comparisons, new-skill knowledge is added to the Ro-SLM system prompt and PRISM compiler.Source: 4) Baselines · pp. 5–6

5) Evaluation:

  • A robot operation succeeds when its state-transition sequence matches ground truth within 1 meter for each position parameter and 4 degrees for yaw.Each transition is represented by [x, y, z, θ], with position changes along the North, East, and Down axes.Source: 5) Evaluation · p. 6
  • Experiments repeat three times and report OSR for partial progress and TSR for full task completion, where TSR is 1 if and only if OSR is 1.Source: 5) Evaluation · p. 6

B. Results

Skill-SLM performs strongly across the UAV task sets, including unseen combinations and newly introduced capabilities. Its performance also holds across SLM families and sizes, with Qwen3-8B strongest overall.

1) Overall Result:

  • Skill-SLM achieves the highest average TSR on all three UAV task sets, and three of four SLMs reach 100.0% TSR and OSR on Seen-Pattern tasks.Seen-Pattern tasks are well represented during training.Source: 1) Overall Result · p. 6
  • On New-Combination tasks, Skill-SLM stays close to Seen-Pattern performance, while PRISM and Ro-SLM degrade substantially.The comparison highlights generalization to unseen combinations of learned capabilities.Source: 1) Overall Result · p. 6
  • On New-Capability tasks, Skill-SLM achieves the highest TSR after new skills are introduced without additional distillation.The results indicate that its compositional reasoning can be extended through skill augmentation.Source: 1) Overall Result · p. 6
  • Skill-SLM performs strongly across SLM families and sizes; larger models perform better within each family, and Qwen3-8B is strongest overall.Qwen models also outperform similarly sized Llama models.Source: 1) Overall Result · p. 6

2) Ablation Study:

The ablations examine skill extraction, distillation, and orchestration. Their results identify operational skill awareness, teacher-generated training, and progressive orchestration as important to reliable task execution.

2) Ablation Study:

  • Direct Skill Extraction falls behind on New-Capability tasks, as expanding the library can introduce inconsistent or redundant skills that hinder selection and composition.The authors identify operational-skill-aware CFG construction as important for reliable reasoning and execution across task scenarios.Source: 2) Ablation Study · pp. 6–7, Table III · p. 7
  • Removing distillation causes the largest degradation, showing that prompts and skill documents alone are insufficient for base SLMs to perform skill-driven reasoning reliably.Teacher-generated examples train task decomposition and skill orchestration.Source: 2) Ablation Study · p. 7, Table III · p. 7
  • One-shot orchestration is overwhelmed by the full plan and skill context, whereas progressive subtask-by-subtask code generation improves coherence and task execution.The one-shot variant often produces correct intermediate operations but fails to maintain coherence for successful execution.Source: 2) Ablation Study · p. 7, Table III · p. 7

C. Ground Vehicle

Ground-vehicle experiments show Skill-SLM generalizing across robot platforms and task sets. The paper also reports a deployment-efficiency trade-off and identifies different favorable model choices depending on memory constraints.

C. Ground Vehicle

  • Skill-SLM performs strongly on ground-vehicle Seen-Pattern, New-Combination, and New-Capability tasks, supporting transfer of skill-based compositional reasoning across platforms.The results are described as consistent with the UAV experiments.Source: C. Ground Vehicle · p. 7

D. Latency and Memory Usage

  • Llama-3.2-3B-Instruct has the highest output throughput at 18.3 tokens/s with 4.3 GB peak memory; Qwen3-4B offers performance comparable to 8B models with less memory and higher throughput.For memory-constrained robots, the paper identifies Llama-3.2-3B-Instruct as the most efficient configuration.Source: D. Latency and Memory Usage · p. 8

V. CONCLUSION

  • The conclusion presents Skill-SLM as a promising approach to reliable, generalizable onboard task planning and robot operation.It cites UAV results against baselines and ground-vehicle results across platforms as support.Source: §V · p. 8
Loading 2610.10812v1…