Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance

• The motivation is that standard reinforcement learning handles short-horizon tasks but typically needs demonstrations or dense rewards for long-horizon skills.Human practice provides an analogy for combining learned primitives without continuous coaching.Source: §1 · p. 1
• BOSS grows an initial repertoire of primitive skills into hundreds of long-horizon tasks without additional human supervision.The agent practices chaining skills, updates its policy, and adds discovered chains to the library.Source: §1 · p. 1, Figure 1 · p. 2
• LLMs guide which skills to chain because random combinations of primitive movements are unlikely to produce meaningful household behaviors.The LLM predicts meaningful next skills from the chain executed so far, while the agent practices those chains in the environment.Source: §1 · pp. 1–2
• BOSS outperforms prior LLM-based planning and unsupervised exploration methods on long-horizon household tasks in novel environments.The evaluation includes simulated ALFRED environments and a real robot.Source: §1 · p. 2

• Skill-based reinforcement-learning methods recombine learned behaviors but most require demonstrations or frequent reward feedback.BOSS is presented as learning long-horizon tasks with minimal human supervision through skill bootstrapping.Source: §2 · p. 2
• Unsupervised reinforcement-learning methods commonly focus on locomotion because diverse behaviors such as running gaits are inherently meaningful.Manipulation-focused methods either require hand-crafted spaces or remain limited to short-horizon skills.Source: §2 · p. 2
• BOSS starts from language-conditioned primitive skills and uses LLM guidance to explore meaningful long-horizon chains among exponentially many possibilities.This targets manipulation tasks that are difficult to discover through unconstrained exploration.Source: §2 · p. 3
• LLM-planning approaches execute open-loop plans with fixed low-level policies, whereas BOSS practices chains in the environment and trains a closed-loop policy.The paper associates this practice with higher success rates for long-horizon execution.Source: §2 · p. 3
• Compared with related LLM-guided exploration methods, BOSS expands a learned skill repertoire and automatically proposes goals in the target environment.This avoids the captioning, hindsight-labeling, or human-designed social-partner requirements described for ELLM, LMA3, and IMAGINE.Source: §2 · p. 3

3 Method

• BOSS comprises base-skill acquisition followed by skill bootstrapping, after which it can execute novel natural-language instructions zero-shot.The method is designed for complex long-horizon tasks with minimal supervision.Source: §3 · p. 3

3.1 Pre-training a Language-Conditioned Skill Policy

• The primitive policy is trained with offline RL on language-annotated trajectories and sparse rewards for primitive-skill completion.BOSS uses IQL and conditions both policy and critic on the trajectory’s language annotation.Source: §3.1 · p. 3

3.2 Skill Bootstrapping

• During bootstrapping, the value function selects an executable initial skill, while the LLM proposes subsequent skills based on the current repertoire and executed chain.Candidate LLM outputs are mapped to existing skills by sentence-embedding nearest neighbors and sampled using token-likelihood distributions.Source: §3.2 · pp. 3–4
• LLM guidance avoids random chaining, whose 20-skill, length-5 search space contains 3.2M possible chains with few likely to be meaningful.The method uses iterative environment practice to learn more robust policies, especially in unseen environments.Source: §3.2 · p. 4
• Completed chains receive sparse rewards for each successful skill, are added to the replay buffer, and are summarized as composite instructions for the growing skill library.Training mixes new bootstrapping data with the original dataset to preserve the initial repertoire.Source: §3.2 · p. 4
• The figure presents two environments: ALFRED, a benchmark for learning agents to follow natural-language instructions for household tasks, and a real-world Jaco arm kitchen-manipulation tabletop environment based on RGB image inputs.Source: Figure 3 · p. 5

4.1 Experimental Setup

• Experiments compare BOSS with unsupervised RL and LLM planners in ALFRED and real-robot kitchen manipulation, using shared primitive-skill pretraining and online bootstrapping.ALFRED uses unseen floorplans and tasks requiring 2–8 primitive skills; the robot setup evaluates unseen object arrangements.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 BOSS Bootstrapping Learns Useful Skills

• BOSS achieves better oracle-normalized return than all nonoracle ALFRED baselines on length-3 and length-4 tasks and is the only method with nonzero success across all task lengths reported.The performance gap is largest on length-4 tasks, while length-6 and length-8 results are omitted because the oracle could not learn them.Source: §4.2 · p. 6, Table 1 · p. 6
• BOSS outperforms SayCan, SayCan+P, CIC, and SayCan+PF by adapting primitive skills and transitions through closed-loop bootstrapping in unseen environments.SayCan+PF reaches 0% success compared with BOSS’s 57%, and most unsuccessful SayCan+P and SayCan+PF trajectories result from policy execution failures.Source: §4.2 · pp. 6–7
• BOSS progressively learns longer skill chains and acquires new skills during bootstrapping, while LLM guidance produces meaningful skill chains and summaries.These analyses directly examine whether bootstrapping expands the repertoire with useful behaviors rather than merely increasing library size.Source: Figure 4 · p. 7, Figure 5 · p. 7
• On the real robot, BOSS matches ProgPrompt on length-2 tasks and is the only method achieving nonzero success on more difficult length-4 tasks.The evaluation covers four tasks in an unseen table arrangement: two of length 2 and two of length 4.Source: §4.2 · p. 7, Table 2 · p. 7

4.2.1 Ablation Studies

• Accurate LLM guidance is increasingly important for longer tasks: weaker OPT-1 guidance degrades performance, random selection performs worse, and BOSS-Rand learns more but less meaningful skills.The comparison shows that skill-library growth alone does not ensure useful behavior; random chaining produces repetitions and meaningless chains.Source: §4.2.1 · pp. 7–8, Table 3 · p. 8

5 Discussion

• BOSS grows a skill repertoire by practicing skill chains with LLM guidance, starting from an initial library and adding learned composite behaviors.The algorithm resets the environment, rolls out skills, proposes subsequent skills, updates the repertoire, and retrains with offline RL.Source: §5 · p. 8, Appendix · p. 13
• BOSS requires environment resets and primitive-skill success detection, uses greedy one-skill-at-a-time chaining, and currently learns only combinations of its initial skills.The paper identifies reset-free RL, longer-horizon proposal mechanisms, and unsupervised low-level skill learning as future directions.Source: §5 · p. 8

A.1.1 Dataset Details

• Environment preparation converts ALFRED data into 73k language-annotated primitive-skill trajectories after separating primitive skills and merging navigation with following skills.The resulting environment uses 12 discrete actions and 82 object types, with sparse rewards for primitive-skill completion.Source: §A.1.1 · p. 14, §A.1.2 · p. 14

A.1.3 Evaluation Tasks

• 40 zero-shot evaluation tasks span 4 unseen ALFRED floor plans, with each task requiring 2–8 primitive skills.Ten tasks are sampled per floor plan from VALID-UNSEEN arrangements and paired with human-written high-level descriptions.Source: §A.1.3 · p. 14, Figure 7 · p. 14

A.2 Language Model Prompts

• The bootstrapping prompt gives the LLM the skill library and seven in-context ALFRED examples to predict a distribution over next skills.A separate prompt summarizes composite skills for labeling before they are added to the library.Source: §A.2 · p. 15, Figure 8 · p. 15

B Training Implementation details and Hyperparameters

• IQL pretrains primitive-skill policies and is reused for fine-tuning, using advantage-weighted regression with critic updates from dataset transitions.The implementation uses shared Q and value backbones with separate heads and language-conditioned inputs.Source: B Training Implementation details and… · pp. 15–16, §B.1 · p. 17

B.1 ALFRED Environment

• ALFRED agents process 300 × 300 RGB observations through ResNet-18 and select among 579 masked action choices formed from 12 actions and 82 object types.Policies and critics use transformer architectures, and fine-tuning uses one policy per floor plan with 15–40 starting skills.Source: §B.1 · pp. 16–17
• Fine-tuning mixes old offline data and newly collected data equally for Oracle, BOSS, and its ablations, while CIC uses a different procedure because its language embeddings are incompatible.Shared hyperparameters include batch size 64, 150 training epochs, learning rate 1e-4, and discount γ 0.97.Source: §B.1 · p. 17

B.2 Real Robot Environment

• The real-robot policy combines third-person and wrist-camera features with robot state and language, producing a 4495-dimensional state input.It predicts 15-action chunks with an LSTM, while robot actions use end-effector displacements and discrete gripper commands at 10Hz.Source: §B.2 · pp. 17–18

B.3 Additional BOSS Implementation Details

• BOSS labels completed skill chains with LLM summaries and adds the resulting composite annotation as a new skill in the library.For example, two primitive annotations can be summarized into a higher-level instruction such as “put a spoon on the counter.”Source: §B.3 · p. 18
• Each rollout contributes attempted-chain data, an exact concatenated primitive-skill label, and trajectories for all executed primitive skills to the replay buffer.The exact label preserves steps that a high-level LLM summary may omit, while primitive trajectories continue refining the original skills.Source: §B.3 · pp. 18–19
• Bootstrapping uses 40 timesteps per primitive skill, limits chains to two skills, and restricts the second skill to primitives while allowing learned composites as initial skills.This makes newly learned chains only one primitive skill longer than the first sampled skill.Source: §B.3 · p. 19
• Prefixing each skill annotation with its extracted main verb improves embedding-based matching by reducing noun-driven substitutions between different actions.The authors report that this change allowed use of the smaller all-mpnet-v2 embedding model in most cases.Source: §B.3 · p. 19
• Skill-bootstrapping experiments use one RTX 3090 with LLaMA-13B 8-bit inference and require roughly 3–5 days, mainly because of simulator speed.Policy pretraining takes around 10 hours on one RTX 3090.Source: §B.3 · p. 19

B.4 CIC Implementation

• CIC uses the same pretraining dataset as BOSS and operates on fixed-length 768-dimensional language embeddings.Its critic and policy architectures were modified accordingly, with CIC-specific latent and rollout settings.Source: §B.4 · pp. 19–20

B.5 SayCan Implementation

• SayCan’s implementation includes in-context task examples that specify ordered low-level actions for household objectives.Examples cover placing objects, cooling or heating items, cleaning utensils, and operating a lamp.Source: §B.5 · p. 20
• SayCan predicts the next skill by choosing from the available skill library given a high-level task and previously executed skills.The prompt frames the interaction as a human-robot conversation and asks for an action sequence.Source: §B.5 · p. 20
• Qualitative zero-shot rollout visualizations compare the plans generated by SayCan+P on household tasks.The visualizations are referenced as examples of SayCan+P plans for the evaluated tasks.Source: Figure 10 · p. 21

B.6 ProgPrompt Implementation

• ProgPrompt represents task descriptions as code by replacing spaces with underscores, then generates commands for execution on the robot.For example, “Pick up the milk” becomes def pick_up_the_milk().Source: §B.6 · p. 21
• Generated pick_and_place(object, object) commands are converted into separate pick and place language instructions before sequential execution.The resulting instructions use the same format as those used to pretrain the policy.Source: §B.6 · p. 21
• Figure 11 shows a BOSS rollout completing all four tasks after skill bootstrapping, while Figure 12 presents example plans from SayCan+P.The SayCan+P caption attributes its errors mainly to policy execution failures.Source: Figure 11 · p. 22, Figure 12 · p. 22
• A soap-placement task is recorded as an execution failure after commands to pick up, wash, and place the soap.The listed sequence first picks up the soap, places it in the sink while operating the water, and then sets it between the sinks.Source: §B.6 · p. 22

C.1 ALFRED Results

• SayCan mostly fails during planning, with planning errors occurring 57.5% of the time, whereas SayCan+P mainly fails during policy execution.The comparison is reported in Table 4.Source: §C.1 · p. 23, Table 4 · p. 23
• SayCan+PF performs similarly to SayCan+P, indicating that naive fine-tuning does not greatly improve the success rate of final plans.This result is reported alongside the Table 4 failure analysis.Source: §C.1 · p. 23
• SayCan+BOSS performs much better than BOSS alone, showing that bootstrapped policies can be combined with high-level SayCan planning.The passage attributes the improvement to robust policies learned through LLM-guided skill bootstrapping.Source: §C.1 · p. 23
• The ALFRED comparison reports evaluation by task length, including length-2, length-3, and length-4 tasks, with return and success metrics.The supplied table structure lists these task-length categories and aggregate measures.Source: §C.1 · p. 23

C.2 Real Robot Results

• The real-robot evaluation contains four tasks in the environment setup shown in Figure 11.Source: §C.2 · p. 23
• The evaluation includes two-step tasks that either place the black bowl in the sink or put it in the dish rack.Both tasks begin by picking up the black bowl.Source: §C.2 · p. 23
• The longer tasks require cleaning the black bowl before placing it in either the dish rack or a gray plate.Each four-step sequence picks up the bowl, puts it in the sink, retrieves it, and places it at the target location.Source: §C.2 · pp. 23–24
• Full returns and success rates are reported for the real-robot evaluation comparisons in Table 5.The supplied material identifies the table as the source of the complete results.Source: §C.2 · p. 24, Table 5 · p. 24