Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skills

1 INTRODUCTION

• Current code-generation methods struggle with noisy long-horizon planning, ambiguous instructions, fixed primitives, and limited context, while repeated feedback or flat-code edits can undermine stable generalization.The introduction motivates reusable skill functions as a more stable way to store feedback knowledge.Source: §1 · p. 2
• Human feedback is presented as a reliable and efficient alternative to LLM-only guidance because people can readily identify and correct robotic-task errors, and code-generation feedback can be tested within seconds.The introduction notes that traditional deep-learning training can take hours or days.Source: §1 · p. 3
• Across simulation and real-world settings, LYRA reaches a 0.93 success rate, up to 27% higher than baselines, and improves correction-round efficiency by 42% versus LLM-based closed-loop methods.The evaluation spans Ravens, Franka Kitchen, MetaWorld, and real-world tasks using a Franka FR3.Source: §1 · p. 3
• LYRA robustly solves the “build a house” task, which requires planning over 20 primitives.The paper identifies this as the first successful solution to the task.Source: §1 · p. 3
• The framework encodes human corrections into skills and extends them through a user-designed curriculum while preserving prior functionality for preference-aligned lifelong learning.Learned skills and examples are stored in external memory; RAG retrieves relevant items, and user hints guide skill selection when retrieval alone is insufficient.Source: §1 · p. 3

2.1 PRELIMINARIES

• A skill is a function that induces a behavior, unlike task-specific code that may call skills; learning means producing better instruction-following behavior as evaluated by a human.Users guide adjustments to skill function details to align its output with their desired behavior.Source: §2.1 · pp. 3–4

2.2 PREFERENCE-ALIGNED SKILL ACQUISITION

• Multi-turn acquisition lets users define a skill and setup, then accept, reject, or refine simulated code; tasks should use 1–4 primitives and vary requirements to test generalization.Users can adjust the proposed skill name and parameters, and the agent modifies its implementation under that reserved name.Source: §2.2 · pp. 4–5

2.3 LIFELONG LEARNING-ORIENTED CAPABILITY EXTENSION

• To address distribution gaps and prompt-induced hallucination or forgetting, a user-designed curriculum extends skills to new cases, rechecks prior tasks, and stores skills and successful instruction-plan examples in external memory for RAG.The agent can create a nested new skill or modularly extend an existing one while preserving its prior functionality.Source: §2.3 · p. 5

2.4 TASK-SPECIFIC-ORIENTED RETRIEVAL AND PLANNING

• With fixed context budgets, the agent retrieves the 10 most similar instruction-plan examples and relevant skill headers, while user-provided hints direct retrieval toward useful learned behaviours.Examples are indexed by instruction and skills by docstring; if a needed sub-behaviour is missing, the user pauses planning to teach it.Source: §2.4 · pp. 5–6
• The framework solves all tasks across over 20 Metaworld and Franka Kitchen variations, handles house-building and Jenga-tower tasks requiring more than 20 planning steps, and completes real-world FR3 temple-building.Source: §2.4 · p. 6

3.1 EXPERIMENT SETUP

• The LYRA w/o memory ablation randomly appended learned skills and examples to the prompt until the context window filled, testing how irrelevant information affects feedback efficiency.In Ravens, it sampled 25 of 49 skills and 25 of 86 examples; in Franka Kitchen and MetaWorld, it sampled half of the learned skills and examples.Source: §3.1 · p. 6
• Evaluation covered customized Ravens, Franka Kitchen, and MetaWorld simulations, plus real-world Franka FR3 deployment, using perception models for open-world object data in simulation.Baselines included open-loop CaP (Liang et al., 2023), feedback-based LoHoRavens (Zhang et al., 2023) and DAHLIA (Meng et al., 2025), and LYRA variants with LLM feedback or without memory.Source: §3.1 · p. 6

3.2 SKILL LEARNING WITH HUMAN-IN-THE-LOOP

• 0.93 average success rate gave LYRA up to 27% improvement over variants, compared with 0.77 for LYRA w/ LLM feedback and 0.66 for LYRA w/o memory.CaP achieved only 0.45 SR on unseen structure-building tasks; the comparison averages success over 20 attempts.Source: Figure 3 · p. 7, §3.2 · p. 7
• Human interventions teach precise, preference-aligned skills, while LLM evaluation can accept incomplete tasks; LYRA reused a learned Jenga-layer skill to build a stable tower.Examples include stacking corner-to-corner and aligning blocks with fixed spacing.Source: Figure 3 · p. 7, §3.2 · p. 7
• 2.75 corrections in Franka Kitchen and 2.55 in MetaWorld made LYRA the lowest-correction framework, with a 42% average efficiency improvement over baselines.Hints let users clarify which skills apply when retrieval fails or returns irrelevant data, reducing unnecessary corrections.Source: §3.2 · pp. 7–8, Figure 4 · p. 8

3.3 CAN YOU BUILD A HOUSE?

• The simulated “build a house” task used 12 skills, including 7 learned specifically for house building, with existing skills inherited and combined through lifelong learning.External storage and retrieval preserve learned skills and examples, preventing catastrophic forgetting and supporting a growing skill tree.Source: §3.3 · p. 8, Figure 5 · p. 8

3.4 REAL-WORLD PERFORMANCE

• On a Franka FR3, LYRA performed house-building, Jenga, and writing tasks lasting over 12 minutes, and completed the unseen “construct a temple” task within five rounds of user guidance.The skills are described as embodiment-agnostic and transferable across heterogeneous robot arms.Source: §3.4 · p. 9, Figure 6 · p. 9

Code generation in embodied AI

• LLM code-generation methods range from Voyager’s extensible skill library to robotic API and ROS pipelines, but open-loop control and limited human verification constrain complex long-horizon tasks.Voyager uses an automatic curriculum and self-verification pipeline Wang et al. (2023); CaP generates Python calls to perception and control APIs Liang et al. (2023), while RoboScript integrates object detection, grasping, and motion planning into a ROS pipeline Chen et al. (2024). LLM evaluators provide task-level feedback Zhang et al. (2023); Meng et al. (2025); Zhi et al. (2025), but may fail on extremely long-horizon tasks and neglect human preferences.Source: §4 · p. 9

Robotic manipulation with language feedback

• Human-feedback methods correct manipulation errors, but narrow correction types, dependence on pretrained policies, and task-specific or overwritten feedback limit generalization and retention.Interactive imitation learning corrects trajectory-level errors Co-Reyes et al. (2018); Chisari et al. (2022); Cui et al. (2023); Lynch et al. (2023); Liu et al. (2023). LMPC uses teacher chat corrections but generalizes only to taught tasks and needs large interactive datasets Liang et al. (2024); PromptBook refines prompts for real-world pick-and-place but is scenario-limited and may forget overwritten prompts Arenas et al. (2024).Source: §4 · p. 9

Large-scale robotic foundation models

• VLA foundation models use end-to-end or hierarchical control, but require massive demonstrations and compute; retrieved non-robot data may also misalign with the robot’s view, state, and goal.The cited models include RT-H Belkhale et al. (2024), OpenVLA Kim et al. (2024; 2025), π0.5 Black et al. (2024); Intelligence et al. (2025), Gemini-Robotics Team et al. (2025), and GR00T N1 Bjorck et al. (2025). RAG methods retrieve non-robot demonstrations Ju et al. (2024); Xu et al. (2024); Kuang et al. (2024), while the paper frames reusable code skills recombined on the fly as an alternative for unstructured environments.Source: §4 · pp. 9–10

5 CONCLUSION

• The framework achieved a 0.93 success rate, up to 27% higher than baselines, and a 42% efficiency improvement in correction rounds across benchmarks and real-world settings.Evaluations covered Ravens, Franka Kitchen, and MetaWorld, as well as real-world tasks.Source: §5 · p. 10
• Skill inheritance, external memory, and hint-guided retrieval encode user feedback as reusable, extendable skills for long-horizon tasks such as building a house.The framework also handled stacking a Jenga tower and writing “ICLR,” and generalized to constructing a temple.Source: §5 · p. 10
• Future work will incorporate advanced multimodal RAG for retrieval and extend the framework to dual-arm collaboration and more complex robotic setups.Source: §5 · p. 10

LLM USAGE STATEMENT

• GPT-4o generated robot-skill code, while ChatGPT was used for sentence-level polishing and grammar correction, which the authors proofread and verified.Source: LLM USAGE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• An anonymous code repository is attached with pre-built trained skill memory for reproducing results; code and data will be open-sourced upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A PSEUDO CODE OF ALGORITHM

• The algorithm retrieves examples using the task and correction, generates code and updates the skill, rolls out the code, and repeats corrections until the task is aligned.It then stores the task-code pair as an example and updates the skill library.Source: A PSEUDO CODE OF ALGORITHM · p. 14

B.1 FRAMEWORK PROMPTING

• The skill-learning prompt supplies the task, target skill, few-shot examples, and useful skills, and asks the agent to solve the task while keeping prior tasks solvable.The system prompt also directs the agent to implement the function and provide flat, task-specific code while preserving the function’s previous functionality.Source: §B.1 · p. 16
• The task-setup prompt translates user instructions into executable environment-setup code using specified types and an available API.Examples show adding repeated red and blue blocks and creating one larger block plus four smaller ones.Source: §B.1 · p. 17
• A skill-parser prompt converts a proposed skill into a descriptive typed function header and docstring, without implementing the function.Similar skill headers can serve as examples, and a separate refinement prompt revises an existing header or implementation with minimal changes and assumptions.Source: §B.1 · p. 17
• A hint-parsing prompt extracts similar task descriptions from a supplied string into a list, while skill learning can draw on few-shot task examples and other useful skills.Source: §B.1 · pp. 16–18
• The framework exposes primitives for inspecting objects and poses, moving the end effector, pick-and-place, and computing bounding boxes or relative points.The pick-and-place primitive places an object on top of the target pose when that pose is occupied.Source: §B.1 · pp. 18–19

B.2 BASELINE MODEL SETUP

• For fair comparison, the authors adapt CaP, LoHoRavens, and DAHLIA implementations with wrappers for direct integration with their APIs.The baselines use state-of-the-art open-source code and language-generation repositories.Source: §B.2 · p. 19
• LYRA w/ LLM feedback uses RGB-D observations for success evaluation and corrective feedback, while relying on LYRA’s learned skills and examples rather than direct LLM skill learning.The LLM reasons about the task, adjusts the plan, and retrieves relevant memory items; direct skill discovery was difficult in robotic manipulation.Source: §B.2 · p. 19
• Without memory, LYRA randomly samples 25 of 49 learned skills and 25 of 86 examples for Ravens tasks; performance drops on complex tasks but remains above baseline models.This setup appends sampled items until the context window is full to simulate catastrophic forgetting in prompt-only human-in-the-loop approaches.Source: §B.2 · p. 19
• Across Ravens, Franka Kitchen, and MetaWorld, all models start with 9 core primitives, and baselines receive few-shot task plans to generalize to unseen scenarios.Ravens plans include block stacking and spatial arrangements; Kitchen and MetaWorld plans include reaching, puck manipulation, and opening or closing articulated objects.Source: §B.2 · pp. 19–20

B.3 RAVEN TASKS SETUP

• The experiment compares success rates across six customized Ravens tasks.The tasks range from foundational spatial skills to more complex construction behaviors.Source: §B.3 · p. 20
• Place the red block next to the blue block without collision; corner-to-corner alignment is not required.The scene contains two large blocks, and the task targets foundational spatial skills.Source: §B.3 · p. 20
• Stack four medium-sized colored blocks into a tower; random positions or orientations still count as success.Precise alignment is not required, and the task provides a base skill for more complex behaviors.Source: §B.3 · p. 20
• Build user-instructed block structures, including unseen 4 × 3 × 3 pyramids, 1 × 3 × 3 walls, and 4 × 4 × 1 bases.The prompt provides 2 × 2 × 2 cube and 3 × 2 × 2 pyramid plans as examples.Source: §B.3 · p. 20
• Make a human face by building a circle and arranging two blocks plus a rectangle as facial features; orientation is unrestricted.The scene contains several medium-sized colored blocks and one rectangular block.Source: §B.3 · p. 20
• Build a stable Jenga tower by stacking multiple rectangular blocks in alternating orientations.The blocks may be uniform or mixed in color.Source: §B.3 · p. 20
• Build a house using yellow blocks for the base, brown blocks for the roof base, and red plates for the roof tiles.The scene contains blocks of various colors and shapes.Source: §B.3 · p. 20

C.1 HARDWARE SETTING

• The real-world demonstrations use a Franka FR3 with a table-edge Intel RealSense D435i camera positioned about 1 m in front of the robot.The setup is illustrated in Fig. S1 and mirrors the simulation camera configuration.Source: §C.1 · p. 21, Figure S1 · p. 21
• The robot base frame is centered at joint 0, with x directed toward the table and z vertically upward from its surface.Source: §C.1 · p. 21
• Real-world pick-and-place uses a Franka two-finger gripper, unlike the simulation benchmark’s suction gripper.Source: §C.1 · p. 21
• Consistent primitive APIs across simulated and real embodiments and gripper types enable skills developed in simulation to transfer directly to the physical robot.Source: §C.1 · p. 21

C.2 PERCEPTION

• Unlike simulation’s predefined APIs for privileged object information, the real-world setup uses vision foundation models to parse task scenes.Source: §C.2 · p. 21
• Grounded SAM 2 Ren et al. (2024), using SAM 2.1.hiera.large and GroundingDINO.swint.ogc, detects target-object masks and bounding boxes for open-world pose estimation.Source: §C.2 · p. 21
• The object center is estimated from the mean of mask pixels projected into 3D, while FoundationPose Wen et al. (2024) and CAD models recover the object frame’s x- and y-axis directions.Comparing angle differences between object and robot-base frames determines the object’s rotation pose.Source: §C.2 · p. 21

C.3 DEPLOYMENT

• The Franka FR3 action space uses six dimensions for end-effector translation and rotation, with customized Cartesian linear planning for precise paths instead of default OMPL-based planning.The setup runs with the official Franka ROS2 library on Ubuntu 22.04 and configures detection tolerances, velocity and acceleration scaling, and orientation/pose constraints; Table S2 summarizes the adopted values.Source: §C.3 · p. 22, Table S2 · p. 22
• The ROS2 deployment script initializes a multithreaded executor and RealWorldEnvironment, resets the environment, and provides a slot for task-specific code and skill calls.The example script uses a `demo(record_cfg, camera_args)` entry point and includes exception handling and node/executor shutdown.Source: §C.3 · pp. 22–23

• Figure S2 depicts block stacking across variations in position, rotation, color, size order, tower shape, and multiple sorted towers.Source: Figure S2 · p. 24
• The agent places red blocks around a blue reference to form a cross and builds varied structures, including cubes, pyramids, walls, and bases.Source: Figure S3 · p. 25, Figure S4 · p. 25
• The smiley-face and Jenga examples show the agent reusing basic skills to assemble a full face and stack a tower.Source: Figure S5 · p. 25, Figure S6 · p. 26
• Franka Kitchen snapshots show kettle pick-place and cabinet or microwave opening, while Meta-World snapshots cover 20 listed tasks.Source: Figure S7 · p. 26, Figure S8 · p. 26
• Table S3 reports correction counts for individual Franka Kitchen and Meta-World tasks, with “-” denoting task failure.Source: Table S3 · p. 26

E BUILD A HOUSE

• The build_house routine requires 14 yellow base blocks and stages the base blocks, roof tiles, roof beam, and roof base at separate workspace edges before construction.Source: E BUILD A HOUSE · p. 27
• At the workspace center, the routine builds the block base and assembles the roof from its base, beam, and tiles.Source: E BUILD A HOUSE · p. 27
• The base layout stacks two blocks at each front corner, leaves a one-block-width gap between corners, extends lines backward, and forms a back wall.Source: E BUILD A HOUSE · p. 28
• Component identification uses red tiles with one dimension under 0.02, a brown beam with a square side and length at least 3 times its width, and a brown roof base with two dimensions at least 10 times the third.Source: E BUILD A HOUSE · p. 28
• Roof assembly places the base, positions the beam above it, and places exactly six tiles in two groups of three.Source: E BUILD A HOUSE · pp. 29–30
• The generic structure helper arranges blocks in three dimensions and assumes the supplied blocks are numerous enough and homogeneous in size.Source: E BUILD A HOUSE · pp. 29–30
• The stacking helper checks whether a block already occupies the start pose, then places blocks sequentially and updates height using each block’s size.The start-position check uses a 0.05 tolerance on each position coordinate.Source: E BUILD A HOUSE · p. 31
• The alignment helper places a block beside a reference along x, -x, y, or -y, calculating the offset from both block sizes and a gap.It applies the reference block’s rotation to the direction and uses the reference pose for the placed block.Source: E BUILD A HOUSE · pp. 31–32

F REAL-WORLD DEMONSTRATIONS

• Real-world demonstrations show tasks including building a house, stacking a Jenga tower, writing “ICLR,” building a temple, and building a human face.Source: Figure S10 · p. 33, Figure S11 · p. 33, Figure S12 · p. 33
• Additional real-world demonstrations show block stacking with corner-to-corner alignment, 0.5 cm shifts, and blocks ordered big-to-small or small-to-big; another depicts baseline models CaP or DAHLIA.Source: Figure S15 · p. 34, Figure S16 · p. 34, Figure S17 · p. 34