PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming

John Woods, Hasti Seifi

arXiv:2608.14944v1cs.ROcs.CLcs.LG

TL;DR

Natural-language robot programming systems struggle with complex, multi-step tasks and often do not learn reusable skills from prior interactions. SkillComposer combines evaluator-guided generation with online macro learning, improving reliability and usability while shortening programs and reducing prompting effort.

  • Problem

    Existing natural-language robot programming systems rely on fixed skills and often require users to decompose complex goals, repair failures, or adapt language to available APIs.

  • Method

    SkillComposer uses a generate-test loop and online library learning to validate robot programs and compress recurring structures into reusable macro skills.

  • Results

    Evaluator-guided generation improved reliability, while macro learning reduced program length and enabled skill reuse; users preferred SkillComposer and needed fewer prompts than with a baseline coding LLM.

  • Takeaways & Limitations

    SkillComposer supports high-level, multi-step robot programming through interfaces that retain, reuse, and refine task knowledge over repeated use.

Abstract

from arXiv · show

Natural-language interfaces can lower the barrier to programming robots, but existing systems struggle when users request complex tasks. While large language models (LLMs) perform well with simple commands, they often struggle to generate code for multi-step tasks, decompose high-level instructions, or reuse prior solutions. We present SkillComposer, an interactive natural-language robot programming system for simulation environments that continually learns reusable program abstractions. SkillComposer uses a generate-test architecture in which an LLM iteratively generates and revises robot programs before execution. Successful programs are stored and processed by an online library-learning algorithm that compresses recurring function sequences into reusable macro skills for future tasks. We evaluate SkillComposer through ablation experiments and a user study with 12 participants to determine its effectiveness on manipulation and robot caregiving tasks. The results show that evaluator-guided generation and learned abstractions improve success rates and usability while reducing user effort in natural-language robot programming.

I. INTRODUCTION

SkillComposer addresses the expertise barrier in simulator-based robot programming by learning reusable skills from generated programs rather than relying on isolated prompts or fixed primitives. Its generate-test architecture and evaluations examine task performance, efficiency, skill reuse, and usability across manipulation, interaction, and assisted-feeding scenarios.

  • Motivation: Robot simulators improve safety and efficiency before physical deployment but still require programming and robotics expertise, limiting access for domain experts.The introduction highlights assistive robotics as a particularly important setting for users who know desired behaviors but lack implementation expertise.
  • Motivation: Natural-language robot programming enables ordinary-language task descriptions, but prior systems typically depend on manually defined primitive skills.These systems can select actions, generate executable programs, and support non-expert users, while still requiring decomposition of complex tasks.
  • SkillComposer: SkillComposer generates, evaluates, and stores successful robot programs, using coder- and evaluator-LLM feedback in a generate-test loop before acceptance.The system learns reusable robot skills from generated programs for future reuse rather than treating each prompt as an isolated generation task.
  • Evaluation: The evaluation studies generate-test and library learning through ablations and a user study spanning manipulation, interaction, and assisted-feeding tasks.RQ1 measures task success, program length, runtime, and macro usage; RQ2 compares usability and user experience with a baseline coding LLM.
  • Contributions: The paper contributes SkillComposer and empirical evidence on how generate-test and online skill learning affect success, efficiency, skill reuse, and usability.The reported scenarios include generic manipulation and assisted feeding.

II. RELATED WORK · A. Natural Language Robot Programming · B. Learning Reusable Robot Skills and Program Abstractions

Related work spans LLM-based natural-language robot programming, reliability-focused generate-test methods, and systems that learn reusable skills or abstractions. SkillComposer differs by learning callable robot skills directly from accepted LLM-generated programs and user interaction history.

  • A. Natural Language Robot Programming: SayCan selects robot actions using language instructions and physical capabilities, while ProgPrompt generates programs from available actions, objects, and example programs.
  • A. Natural Language Robot Programming: ChatGPT for Robotics generates robot code and task plans from a robot API, while GenSim stores high-quality generations for later fine-tuning and evaluation.
  • A. Natural Language Robot Programming: Related systems improve LLM-generated robot-code reliability by evaluating task correctness beyond syntactic validity and iteratively refining programs through analyzer feedback and visual input.
  • B. Learning Reusable Robot Skills and Program Abstractions: LATM and CREATOR prompt LLMs to generate callable tools, whereas Voyager builds executable skills continually through interaction with Minecraft.
  • B. Learning Reusable Robot Skills and Program Abstractions: In robotics and programming languages, prior work connects reusable-skill learning with wake-sleep methods, library learning, and language-model-assisted abstraction discovery.
  • B. Learning Reusable Robot Skills and Program Abstractions: SkillComposer learns reusable robot skills directly from accepted LLM-generated robot programs, expanding the callable skill set through user interaction history without manual skill design.
  • B. Learning Reusable Robot Skills and Program Abstractions: DreamCoder learns reusable library functions from program corpora, while Stitch discovers abstractions that minimize program corpus size.

III. SKILLCOMPOSER · A. Generate-Test Loop

SkillComposer combines an interactive generate-test loop for producing executable robot programs with online macro learning that discovers reusable skills from accepted programs. The loop uses evaluator and static validation, revises failed candidates for up to five iterations, and executes only programs accepted by both stages.

  • III. SKILLCOMPOSER: SkillComposer combines a generate-test loop for executable robot-program generation with online macro learning that discovers reusable skills from previously accepted programs.Users interact through a locally hosted Gradio interface supporting natural-language prompting and manual robot and camera interaction.
  • A. Generate-Test Loop: The Generate-Test Loop uses a coder LLM to generate candidate robot programs and a two-stage pipeline to check functional and structural validity before execution.The two stages are evaluator-LLM assessment followed by static validation.
  • A. Generate-Test Loop: The coder LLM converts each user instruction into a candidate program using available function schemas and JSON function calls with named arguments.The initial primitive functions are get_info, move_to_object, move_to_position, grasp_object, and release_object.
  • A. Generate-Test Loop: The evaluator LLM judges whether a candidate satisfies the user’s request and returns natural-language feedback when it identifies a suspected problem.Programs that pass evaluator review proceed to static validation.
  • A. Generate-Test Loop: The static validator checks that a program is well-formed, uses available-schema functions, supplies valid arguments, and includes required structure.This stage follows evaluator-LLM assessment before execution.
  • A. Generate-Test Loop: Accepted programs pass both evaluation stages and are executed in RCareWorld before being sent to the macro learning module.Failure at either stage prevents execution and triggers revision.
  • A. Generate-Test Loop: When either stage fails, its feedback is appended to the coder-LLM conversation, prompting program revision rather than execution.The system repeats this generate-and-revise process for up to five iterations.
  • A. Generate-Test Loop: After five unsuccessful iterations, SkillComposer reports an error to the user instead of executing an invalid program.This termination condition applies when the candidate never passes both evaluation stages.

B. Online Macro Learning

SkillComposer adapts Stitch for online learning by converting accepted robot programs into symbolic expressions and compressing repeated action sequences into reusable macro skills. It filters abstractions for executable imperative robot programs, exposes valid macros through function schemas, and expands them back into primitive calls before execution.

  • Generating Macro Candidates with Stitch: Accepted JSON programs are converted into s-expressions representing primitive calls and ordered execution, then appended to a corpus of previously generated programs.Primitive arguments follow their function schemas, while sequential execution uses a right-nested then structure.
  • Generating Macro Candidates with Stitch: Stitch searches the corpus for abstractions that reduce its rewritten size, identifying repeated robot-function sequences that can become higher-level macro skills.SkillComposer runs Stitch for up to 20 iterations with maximum arity five.
  • Substitution and Filtering: SkillComposer expands and filters Stitch abstractions so valid macros contain at least two primitive calls, binary then sequencing, known primitives, and variables only in primitive argument positions.The system also introduces fresh variables for missing arguments and renames variables consistently to deduplicate abstractions.
  • Substitution and Filtering: The reimplemented learning loop selects the highest-ranked valid candidate, rewrites the corpus, and repeats until ten valid abstractions are learned or no valid candidates remain.The macro set is relearned after each accepted prompt so available skills reflect the full interaction history.
  • Using Learned Macros: Each valid abstraction becomes a function schema available alongside primitive functions, allowing future LLM-generated programs to call learned macros that are expanded into primitive robot calls before execution.Schemas include names, descriptions, parameters, types, and parameter descriptions inferred from abstraction variables and documented with contextual examples.

IV. EXPERIMENTS

The experiments assess SkillComposer through complementary ablation and within-subjects studies targeting task performance and interactive user experience. The ablations isolate generate-test and online macro learning, while the user study compares SkillComposer with a baseline coding LLM.

  • Experimental design: The evaluation comprises two complementary studies addressing research questions RQ1 and RQ2.RQ1 concerns task performance, while RQ2 concerns user experience in interactive robot programming.
  • Ablation experiments: Ablation experiments isolate the effects of the generate-test loop and online macro learning on task performance.These experiments address RQ1.
  • User study: A within-subjects study compares SkillComposer with a baseline coding LLM in an interactive robot programming setting.This study addresses RQ2 by evaluating user experience.

A. Ablation Experiments

The ablation experiments compare four SkillComposer variants to isolate the contributions of evaluator-guided revision and online macro learning. They evaluate these variants across manipulation and caregiving tasks using success, complexity, macro-use, and runtime measures.

  • System Variants: Four variants isolate direct coding, evaluator-guided generate-test revision, online macro learning, and their combination in SkillComposer.The combined system uses both the generate-test loop and online macro learning.
  • Evaluation Tasks: Each variant is tested in Objects and Feeding environments with 20 natural-language prompts per environment.Objects covers manipulation, spatial reasoning, multi-step action planning, and goal planning; Feeding covers simple and multi-step assistive caregiving interactions.
  • Evaluation Measures: The experiments measure success rate, program length, macro usage, and runtime as dependent variables.Success indicates task completion; program length counts final function calls, macro usage measures the percentage of program length comprising macro calls, and runtime includes applicable generate-test iterations.

B. User Study

The user study compared SkillComposer with a baseline coding LLM in a within-subjects design involving 12 novice-oriented participants. Participants completed natural-language robot programming tasks and provided objective, subjective, and interview-based feedback.

  • Study design: The within-subjects study compared full SkillComposer, including generate-test and online macro learning, with a baseline coding LLM using an identical interface and counterbalanced backend order.The design aimed to reduce order effects while isolating the full SkillComposer system from the baseline coder LLM.
  • Participants: 12 participants—3 female and 9 male, aged 23–67—were recruited with none or beginner-level robotics or programming experience.Participants had upper-intermediate English proficiency and normal or corrected-to-normal vision; the criteria targeted novice end users.
  • Procedure: Each approximately 75-minute session included study orientation, consent, a background questionnaire, and a practice task moving a banana 25 cm to the right.The practice task took place in the Bananas environment to familiarize participants with the interface.
  • Tasks and measures: For both systems, participants completed three Feeding-environment tasks: rearranging table objects, preparing a meal, and feeding the care recipient.Participants wrote their own natural-language prompts and could submit as many prompts as desired.
  • Tasks and measures: The study collected objective and subjective measures, followed by interviews about system preference, helpful or confusing aspects, behavior reuse, learned-skill meaningfulness, prompting strategies, task performance, and improvements.The objective measures included prompts used and task success rate, with the passage continuing beyond the supplied excerpt.

V. RESULTS · A. Ablation Experiments

Ablation experiments found that evaluator-guided generate-test systems achieved the strongest success rates, while macro learning mainly reduced program complexity. The full SkillComposer system combined comparable performance with shorter programs and increasing reuse of learned environment-specific macros, at the cost of substantially higher latency from iterative evaluation and revision.

  • A. Ablation Experiments: The generate-test loop achieved the largest success-rate improvement among the ablations.Overall, the generate-test system achieved the highest success rate.
  • A. Ablation Experiments: Evaluator-guided systems outperformed direct code generation across most prompt categories, while all variants performed well on basic object manipulation.The basic-manipulation exception suggests direct LLM generation was often sufficient for simple commands.
  • A. Ablation Experiments: Generate-test increased generation time by approximately four to five times because it added evaluation and revision steps.This was the largest latency cost observed in the ablation experiments.
  • A. Ablation Experiments: Macro learning primarily reduced program complexity by compressing repeated low-level action sequences into higher-level function calls.Systems with macro learning generated shorter programs on average.
  • A. Ablation Experiments: Macro learning alone did not substantially improve success over the baseline and was most effective when paired with the generate-test loop.The full system delivered comparable performance with more compact programs and visible macro reuse.
  • A. Ablation Experiments: Macro availability generally increased as accepted programs accumulated in the corpus, and macro usage tended to rise as more learned skills became available.Figure 5 tracks these trends across 20 prompts in each environment and across five runs.
  • A. Ablation Experiments: Learned macros captured recurring action patterns, including move_and_place_object in Objects and place_object_on_plate and hand_object_to_user in Feeding.These examples demonstrate environment-specific reusable skills.

B. User Study

In the user study, SkillComposer generally outperformed the baseline in task efficiency, usability, and user preference, although the baseline performed slightly better on Feed Care Recipient. Participants valued SkillComposer’s support for high-level, multi-step instructions but suggested improvements in accuracy, response time, and interface design.

  • Task-level results: SkillComposer matched or exceeded baseline task completion rates on Rearrange Objects and Meal Preparation while requiring fewer prompts across all three tasks.It also reduced average completion time on Rearrange Objects and Meal Preparation; Feed Care Recipient was the exception, with the baseline achieving slightly higher success and lower completion time.
  • Task-level results: The largest task-level difference occurred on Rearrange Objects, where the baseline had substantially lower task success as participants issued high-level, multi-object instructions.Participants adapted the baseline to step-by-step prompts over time, reducing the performance gap on the next two tasks.
  • Subjective ratings: 10.7 points: SkillComposer’s mean SUS score exceeded the baseline’s, with scores of 72.1 (σ = 20.1) and 61.4 (σ = 24.6), respectively.SkillComposer’s SUS result was categorized as good usability, while the baseline was categorized as OK usability; the effect was moderate (Cohen’s d = 0.46).
  • Qualitative feedback: 11 of 12 participants preferred SkillComposer, citing stronger support for high-level, multi-step instructions and lower effort.One participant contrasted using a single prompt in SkillComposer with needing six or seven prompts in the baseline.
  • Future improvements: Participants requested improved accuracy and response time, broader applications, and interface changes including voice interactions, a higher-frame-rate preview, and clearer status notifications.Suggested applications included household humanoid robots retrieving objects and bimanual tasks such as pouring wine into a cup.

VI. CONCLUSION

SkillComposer combines evaluator-guided code generation with online skill learning for interactive natural-language robot programming. Validating programs before execution and learning reusable macros enables the robot skill set to evolve over time, while ablations showed improved reliability from the generate-test loop.

  • System overview: SkillComposer combines evaluator-guided code generation with online skill learning in an interactive natural-language robot programming system.The system validates generated programs before execution and learns reusable macros from accepted programs.
  • Online skill learning: Validated programs and learned reusable macros allow SkillComposer’s available robot skill set to evolve over time rather than remain fixed.Macros are learned online from accepted programs.
  • Evaluation: Ablation experiments showed that the generate-test loop substantially improved reliability.The loop validates generated programs before execution.
Loading 2608.14944v1…