Skill Composition for Legged Robot Reinforcement Learning

• Strong individual robot skills still leave open how to execute complex tasks spanning several skills.Source: §1 · p. 1
• End-to-end policies must balance stability, progress, and safety, face long-horizon credit assignment, and cannot reuse competencies without retraining.Competencies are entangled in the same network, so they cannot be studied in isolation.Source: §1 · p. 1
• Separate policies make each skill inexpensive to train, reliable to converge, and independently verifiable.Source: §1 · p. 1
• Freezing trained experts lets the repertoire grow by addition, whereas finetuning sacrifices the modularity motivating decomposition.Source: §1 · p. 1
• Blending continuously combines weighted expert actions, while bridging switches named experts and needs a handover behavior.The decision layer is separate from the experts in both mechanisms.Source: §1 · p. 1
• Arbitrary external switches can hand control to a state outside the incoming skill's initiation set.Source: §1 · p. 2

2. Problem definition

• Each regime has an expert optimized for its own reward, and a separate decision layer and composition operator must solve the original task while retaining expert reusability.The central question is how to learn the experts and composition so performance recovers or exceeds an end-to-end solution.Source: §2 · p. 2

2.1. Two ways of composing experts

• Blending weights concurrently active experts and merges their outputs into one action, with continuous weight changes absorbed by the merge.It is suited to superposed competences such as locomotion and manipulation, though the merge is not guaranteed competent wherever each expert is competent alone.Source: §2.1 · p. 2
• Bridging selects one active expert and inserts a transient connective behavior at a switch to prepare the incoming expert from the outgoing expert’s resulting state.The bridge is responsible for enabling handover, not for solving either regime’s reward.Source: §2.1 · p. 2

2.2. What a good composition method would enable

• Blending could combine skills into behaviors neither expert has alone, while reliable bridges could support long-horizon sequences and high-level instruction.Under either mechanism, the library can grow by training one policy and connecting it to existing skills; bridging could also make inspectable decision layers actionable and provide a place to state and check safety conditions.Source: §2.2 · pp. 2–3

Scope of prior work

• The reviewed bridging literature assumes the outgoing skill may terminate naturally, rather than being interrupted in an arbitrary state.Thus it does not address the unpredictable state inherited by a bridge after interruption.Source: §3 · p. 3

Learning the bridge

• Learned-bridge approaches train transition policies, shape bridge behavior near the next skill’s usual start, or reshape rewards and replay boundary failures,,.Lee et al. add a state-promise estimate because feedback arrives only after the next skill returns a reward.Source: §3 · p. 3

Adjusting skills to fit

• Skill-adjustment methods fit skills together or learn whether one can follow another, while residual learning adds a trained correction over frozen experts,,, [14],.This line reports the strongest published long-horizon manipulation results, at the cost of relaxing expert independence.Source: §3 · p. 3

Choosing the handover moment

• Handover-timing methods include setup policies for terrain-specific skills, driving to preferred manipulation starting positions, and learned quadruped gait-switch criteria, [17],.The cited biped setup policies reuse the target skill’s training-time state assessment, discounting it where it is unreliable.Source: §3 · p. 3

Recent humanoid work

• Recent humanoid approaches blend frozen primitives or experts trained under stochastic interruption, while retrieval-based work matches robot states to motion-database frames [19], [20], [21].These represent two of the recent humanoid lines described in the review.Source: §3 · p. 3

Unresolved gap

• No existing method combines frozen experts, bridging, and arbitrary-time switching; prior handover methods choose when to stop bridging, not where to land in the next skill’s initiation set.The section identifies landing-state selection as the more important unresolved gap.Source: §3 · p. 3

Choosing where to land

• A bridge that chooses only when to hand over cannot aim for a different acceptable landing state that better exploits the robot’s momentum.The jumping-while-running example contrasts initiation states that discard built-up motion with those that exploit it.Source: §4 · p. 3

Comparing and predicting outcomes

• Choosing a landing state requires both a way to compare states by the outcomes a shared behavior produces and a compact expert descriptor predicting near-term behavior after takeover.The proposed descriptors are computed once during expert training and can be consulted by any bridge.Source: §4 · p. 3

Organizing bridges

• It remains open whether one bridge conditioned on the target expert can match specialized bridges while sharing general skills such as balance or momentum shedding.Existing methods use one bridge per skill or pair, which may not scale and forgoes sharing.Source: §4 · p. 3

Interruption

• A decision layer reacting to the world may interrupt a skill mid-run, precisely when the state handed to the bridge is least predictable.Existing work does not cover this interruption case.Source: §4 · p. 3

Trustworthy blending

• A blend is not guaranteed competent where both experts are competent alone, especially when their action spaces overlap and their commands may interfere.The open challenge is to characterize trustworthy blend states without retraining the experts.Source: §4 · pp. 3–4

Measuring transitions

• Long-horizon success counts conflate transition quality with skill and decision-layer quality, motivating separate measures of handover success, time and motion cost, and displacement.Such transition-focused measures would make results more comparable across methods.Source: §4 · p. 4

5.1. Blending: kicking as a residual on locomotion

• For ball-kicking, a learned orchestrator blends frozen locomotion and kicking experts, favoring the kick near the ball without imposing that rule.The kicking policy is trained as a residual on locomotion; locomotion remains frozen.Source: §5.1 · p. 4
• The section presents the blend as a working implementation and baseline, with no quantitative results reported at this stage.The kick is divided into four stages in Figure 1.Source: §5.1 · p. 4, Figure 1 · p. 4
• A separate switch from locomotion to kicking visibly breaks when the kicking skill, trained from standing starts, is invoked mid-stride.The mid-stride state lies outside the kicking skill’s training distribution.Source: §5.1 · p. 4

5.2. Bridging: jumping while walking, and other testbeds

• The bridge carries a walking humanoid into a jump-entry state outside the jumping skill’s narrow training set while retaining momentum.The figure contrasts the transition with the nominal jump execution from trained-on states.Source: §5.2 · p. 4, Figure 2 · p. 5
• A single bridge is trained on recorded-motion gaps, scored on its handover state and subsequent behavior, and does not require retraining when the skill set changes.The target entry state is currently chosen from a small fixed set per skill.Source: §5.2 · pp. 4–5
• The cart-pole testbed shows the bridge reaching a state from which the balancer succeeds, while the differential-drive testbed requires shedding speed before a tight turn.The cart-pole handover is timing-sensitive; the differential-drive handover is speed-sensitive.Source: §5.2 · p. 5