Progressively Learning Heterogeneous Skills in a Unified Latent Space

Motivation

• Physics-based character control lacks a unified executable skill representation that can consolidate heterogeneous behaviors and transfer them efficiently to downstream tasks.Existing methods often rely on task-specific controllers or narrowly curated datasets, limiting reusability and compositionality.Source: §1 · p. 1

Contribution

• HetSkills progressively learns motion tracking, text-to-motion generation, motion completion, and downstream task adaptation within one shared latent representation.The latent space acts as a common executable interface for skill acquisition, composition, and reuse.Source: §1 · pp. 1–2

Method

• The framework uses part-wise character decomposition so body parts receive specialized yet coordinated control signals within a common latent architecture.This design is intended to improve compositional control and generalization.Source: §1 · p. 2
• The tracking skill establishes a shared motion decoder that subsequent tasks can reuse without retraining separate controllers.The tracking foundation provides the motion-control substrate for later skills.Source: §1 · p. 2

Results

• 96.9% success under standard initialization and 81.7% from a neutral pose demonstrate robust text-to-motion generation across initialization conditions.The reported results also cover stable heterogeneous-skill accumulation and compositional control.Source: §1 · p. 2

Physics-based Motion Imitation and Skill Learning

• Physics-based character control traditionally uses motion-capture data and reinforcement learning to produce physically plausible behaviors.Foundational tracking policies imitate reference motion clips, while adversarial approaches encourage stylistic realism without explicit motion matching.Source: §2 · p. 2

Large-scale Motion Priors and Language-conditioned Control

• Large-scale motion priors distill motor skills into universal physics-based latent spaces, allowing diverse downstream tasks to reuse shared motor representations.Later methods combine motion tracking with richer conditioning signals.Source: §2 · p. 2

Part-Wise Motion Learning

• Part-wise structure is used to improve controllability, diversity, compositionality, and interaction-rich behavior synthesis.Physics-based methods use it to compose partial motion priors, decouple imitation objectives, or facilitate part-wise planning.Source: §2 · p. 2

HetSkills

• HetSkills progressively integrates motion tracking, text-to-motion, motion completion, and downstream adaptation within a unified latent space.Its architecture reuses shared state and motion-decoding modules after tracking training.Source: Figure 2 · p. 3

Limitations of Prior Work

• Existing skill spaces are effective for composition under a fixed interface but less suited to progressively accommodating heterogeneous skill types, supervision signals, and conditioning modalities.This limitation motivates a more flexible unified representation.Source: §2 · p. 3

Downstream Task Adaptation with Motion Priors

• Freezing pretrained motion priors and training high-level policies in their latent spaces enables downstream tasks to inherit motion naturalness without retraining low-level controllers.Token-based adaptation methods further target complex or contact-rich scenarios.Source: §2 · p. 3

MDP Formulation

• Physics-based character control is formulated as a goal-conditioned Markov decision process with state s_t, goal g_t, action a_t, transition dynamics, and reward r_t.The policy acts from the current state and goal while maximizing expected discounted return.Source: §3 · p. 3

Action Execution

• Actions are target joint rotations converted into torques through proportional-derivative controllers in the physics simulator.This specifies how policy outputs become physical control signals.Source: §3 · p. 3

Heterogeneous Goals

• Goal signals differ across motion tracking, text-to-motion generation, motion completion, and downstream adaptation in modality, supervision, and data distribution.They include future poses, language descriptions, sparse temporal constraints, and task-specific observations or language conditions.Source: §3 · p. 3

Unified Control Interface

• Each skill predicts a latent z_t from its own goal specification, and a shared decoder maps z_t and s_t to the final physical action.The latent space must incorporate new skills while preserving earlier behaviors and enabling downstream transfer.Source: §3 · pp. 3–4

Unified Latent Interface

• HetSkills incorporates skills learned from different data sources, supervision forms, and training stages into a common control representation.The unified latent space is explicitly designed for skill acquisition, composition, and reuse.Source: §4 · p. 4

Tracking Foundation

• The tracking skill simultaneously constructs the unified latent space and produces the shared motion decoder reused across subsequent stages.It also uses coordinated body partitions with dedicated latent commands within a common architecture.Source: §4 · p. 4

Progressive Skill Learning

• Subsequent skills introduce distinct goal-conditioning mechanisms over the fixed control substrate.Text-to-motion uses Motion Intuition Distillation to ground language descriptions into latent motion commands, while motion completion handles sparse or partial observations.Source: §4 · p. 4

Architecture

• The tracking skill factorizes control into arm and body branches that predict compact latent commands decoded into joint-level target actions.A shared state encoder captures full-body context, while part-wise controllers specialize in the two partitions.Source: §5 · p. 4, §5.2 · p. 5

Decoder reuse

• After training, the shared state encoder and part-wise controllers are frozen and reused as the common motion decoder across later skills.This preserves a common control space without relearning low-level dynamics or requiring an additional distillation step.Source: Figure 3 · p. 4, §5 · p. 4

Inputs

• The policy receives proprioception, previous actions, and future reference poses represented by target positions, rotations, and velocity differences.Future goals include K target poses, with relative positions and pose errors expressed in the current root frame.Source: §5.1 · pp. 4–5

Part-wise architecture

• The two-part decomposition specializes upper-limb behaviors while maintaining coordinated whole-body behavior through a shared state representation.The arm branch is motivated by the distinct high-frequency, low-inertia dynamics of gesturing, punching, and object interaction.Source: §5.2 · p. 5

Latent space

• A deterministic, compact latent representation separates motion categories such as locomotion, gesture, and acrobatics without explicit category labels.The learned structure is intended to support later skill reuse and composition.Source: Figure 5 · p. 5, §5.2 · p. 6

Skill design

• The text-to-motion skill maps natural-language descriptions to arm and body latent actions while preserving the tracking decoder.Only the text-conditioned mapping is learned; the low-level controller remains frozen.Source: §6 · p. 6

Motion Intuition Distillation

• Motion Intuition Distillation removes dependence on future motion and standard initialization, requiring inference from language, current state, and imperfect history.Reference State Initialization samples an episode’s initial state from any reference-clip frame with probability p_rsi = 0.7.Source: §6.1 · p. 7

Shortcut issue

• Shortcut pathways can let text-conditioned models bypass meaningful language semantics when future motion or privileged information is accessible.Nearby future frames and matched initial states provide easier predictive routes than semantic inference.Source: §6.1 · p. 7

Architecture

• The skill uses a frozen TMR text encoder and two part-wise Transformer encoders to predict latent actions decoded into physical actions.The inputs include proprioception, the previous action, text embedding, and recent observation history.Source: §6.2 · p. 7

Shared decoder reuse

• Freezing the shared decoder reduces text-to-motion training to latent prediction and allows comparatively small text-annotated data to build on a diverse motion prior.The decoder supplies already learned low-level control rather than jointly learning physical control again.Source: §6.2 · p. 7

Unified formulation

• Motion completion recovers physically plausible full-body motion from sparse or partial observations across heterogeneous conditioning signals.The framework treats VR tracking, motion in-betweening, and human-scene interaction under a common formulation.Source: §7 · p. 8
• A single skill family maps current proprioception and sparse target observations to latent actions compatible with the shared decoder.This avoids introducing a separate controller for each conditioning type.Source: §7 · p. 8, §7.1 · p. 8

VR Tracking

• VR tracking provides the next-frame head and hand states, requiring the controller to infer the remaining full-body motion.It represents a spatially sparse observation setting and is trained with behavior cloning using the stage-1 tracker as teacher.Source: §7.1 · p. 8

Motion In-betweening

• Motion in-betweening supplies a future full-body pose and time offset, with target frames sampled 5 to 30 frames ahead.The controller synthesizes plausible intermediate motion before resampling another future target.Source: §7.1 · p. 8

Human-scene Interaction

• Human-scene interaction uses future target interaction states and time offsets to model motions involving everyday objects such as chairs and sofas.The SAMP dataset lies outside the tracker’s training-data distribution.Source: §7.1 · p. 8, Figure 7 · p. 8

• HetSkills reuses the language-conditioned motion distribution for new objectives without retraining the motion prior or low-level controllers.Source: §8 · p. 9

Standardized Task Description

• A standardized task unit represents initialization, execution condition, and termination, allowing heterogeneous skills to be specified, organized, composed, and extended consistently.Source: §8.1 · p. 9

Task Composition

• The shared interface supports independent execution and sequential organization of language-conditioned motion, interaction, and goal-conditioned locomotion skills for long-horizon composition.Source: §8.1 · p. 9

Compositional Task Guidance

• Multiple part-wise language priors are routed according to task context and refined with task-conditioned residuals to represent complex or changing downstream behaviors.Source: §8.2 · p. 9

Optimization

• Downstream adaptation trains lightweight guidance modules with PPO while keeping pretrained priors and decoders fixed.The learned modules include part-wise gating and residual networks, together with a value network.Source: §8.3 · p. 10
• Latent magnitude and temporal smoothness penalties counter task-reward-driven drift, keeping adapted actions stable and improving motion naturalness and robustness.Source: §8.3 · pp. 9–10

• Experiments use Isaac Lab with ProtoMotions, 120 Hz physics simulation, and 30 Hz policy execution, training across four progressive stages on two RTX 5090 GPUs.Source: §9 · p. 10
• Progressive training uses heterogeneous data sources without requiring a single annotation format, allowing each stage to use task-appropriate data.Source: §9.1 · p. 10

Tracking and Motion Completion

• Motion tracking uses filtered AMASS sequences, with the cleaned set also reused for VR tracking and motion in-betweening.Filtering removes non-physical artifacts including limb penetration, body floating, and interactions with unmodeled objects.Source: §9.1 · p. 10

Downstream Tasks

• Downstream evaluation covers online path following, target striking with hands or legs, and staged pick-and-place behavior without task-specific demonstrations.The path-follow task uses randomly generated paths through complex terrain, while strike targets are randomly placed.Source: §9.2 · pp. 10–11, Figure 8 · p. 11

Tracking and Motion Completion

• Tracking and motion completion are evaluated with success rate and MPJPE across AMASS training and test splits and sparse spatial or temporal conditions.Source: §9.3 · p. 11

Text-to-Motion

• Text-to-motion evaluation measures pose-level robustness under ground-truth-first-frame and neutral-pose initialization, plus semantic alignment using HumanML3D retrieval.Source: §9.3 · p. 11

Downstream Task Adaptation

• Downstream adaptation is evaluated on the three designed tasks against relevant baselines, with language instructions specified separately.Source: §9.3 · pp. 11–12

Tracking and Motion Completion

• Qualitative tracking examples assess whether the shared decoder reproduces diverse reference motions with stable balance, plausible contacts, and coordinated full-body control.Source: Figure 9 · p. 12

Tracking

• HetSkills-1step achieves the highest success rate under fair inference, while duplicating one observed frame provides nearly the same performance as the two-frame variant.The two variants differ because tracking was trained with a two-frame future window; HetSkills-1step duplicates a single frame at inference.Source: §10.1 · p. 12
• 42.3 mm to 35.5 mm: increasing part granularity from one to five parts reduces test-set MPJPE, but adds computational overhead and reduces semantic interpretability.The authors therefore adopt a two-part arm-and-body decomposition as a balance between accuracy and efficiency.Source: §10.1 · p. 12

Motion Completion

• 99.9% and 97.8%: HetSkills succeeds on training and test splits for VR-driven motion completion from head-and-hand observations.The lower body is underdetermined by only three upper-body endpoints, so test-set positional errors may not reflect motion quality.Source: §10.1 · pp. 12–13
• 99.9% training and 100% test success: HetSkills performs motion in-betweening on AMASS, while achieving 96.1% on human-scene interaction.These results support compatibility among skills trained on different motion distributions and conditioning modalities, although success rate does not fully capture plausible intermediate trajectories or naturalness.Source: §10.1 · p. 13

Text-to-Motion

• 96.9% and 81.7%: HetSkills outperforms MaskedMimic and CLoSD under first-frame and neutral-pose initialization for text-to-motion generation.Removing RMI reduces neutral-pose performance, while adding a residual branch beside the text encoder fails under both protocols.Source: §10.2 · pp. 13–14
• HetSkills preserves strong text-motion alignment under neutral-pose initialization, with higher R@N and lower MedR than the baselines.These retrieval metrics complement MPJPE-based success rates by evaluating whether generated motion remains recognizable as the intended textual action.Source: §10.2 · pp. 13–14

Downstream Task Adaptation

• Semantic guidance helps downstream policies preserve natural motion and explore task-relevant latent regions, enabling reliable convergence on coordinated pick-up.Without semantic guidance, the sparse bimanual-lifting region is difficult to discover within the limited sample budget.Source: §10.3 · p. 15
• The MoE module dynamically selects and combines language priors according to task context, including complementary instructions during low-activity phases.This supports smooth transitions and coherent long-horizon behavior across heterogeneous task stages.Source: §10.3 · p. 15

• Current experiments do not fully cover larger-scale skill accumulation, more complex long-horizon composition, or open-ended human-object and human-scene interactions.Source: §11 · pp. 16–17
• Text-driven motion generation still struggles with high-difficulty actions and ambiguous language descriptions.Motion Intuition Distillation improves robustness to different initial states and motion histories, but annotation quality remains limiting.Source: §11 · p. 17
• Semantically similar descriptions can correspond to substantially different motions, making fine-grained language-motion alignment difficult.Source: §11 · p. 17
• Future work proposes more diverse contact-rich and scene-aware data, contact-aware modules, higher-quality text-motion annotations, broader skills, and real-world robot control.Source: §11 · p. 17
• More robust task decomposition, automatic skill selection, failure recovery, and language-model-based high-level planning are proposed for open-ended long-horizon tasks.The current work uses an LLM to help organize high-level instructions into task sequences.Source: §11 · p. 17

• HetSkills extends one shared control substrate with skills from different data sources, supervision forms, and training stages rather than dedicated controllers.Source: §12 · p. 17
• The framework integrates motion tracking, text-to-motion generation, motion completion, and downstream task adaptation.Source: §12 · p. 17
• Natural language serves as an interface for both motion generation and task guidance.Source: §12 · p. 17
• Part-wise latent control, motion intuition distillation, and language-guided latent composition support motion preservation, skill reuse, and adaptation without task-specific demonstrations or motion-prior retraining.Source: §12 · p. 17

Tracking skill

• Tracking uses PPO with a 512-dimensional shared state encoder, final-action Gaussian exploration, a 0.5 m termination threshold, and RSI probability 0.8.Source: A Implementation Details · p. 19

Text-to-motion skill

• Text-to-motion uses separate arm and body Transformer branches with current-state, language, and six history tokens from a 60-step window.Source: A Implementation Details · p. 19

Motion completion skill

• Motion completion covers VR tracking, motion in-betweening, and human-scene interaction using sparse-goal conditioning.Source: A Implementation Details · p. 19

Downstream adaptation

• Downstream adaptation freezes the pretrained motion prior and low-level decoder, optimizing only task-guidance modules with 64-dimensional arm and body residual latents.Source: A Implementation Details · p. 19

Progressive training

• Later stages train independently on the shared latent space, reusing the frozen low-level controller instead of jointly optimizing one monolithic model.Source: B Training Time and Compute · p. 19

Training cost

• Each downstream task can be trained within 5 hours on one RTX 5090 GPU, while instruction branches are batched in parallel.Source: B Training Time and Compute · p. 19

Reward design

• Downstream rewards encourage task completion while regularizing motion with energy penalties to remain physically plausible and compatible with the pretrained prior.Source: C Downstream Task Rewards and Settings · p. 19

Path Follow

• Path-following rewards track online-generated horizontal target paths using root position, leg energy, speed, acceleration, and termination constraints.The path uses maximum speed 5 m/s, maximum acceleration 2 m/s2, and terminates beyond 1.5 m horizontal error.Source: C Downstream Task Rewards and Settings · pp. 19–20

Strike

• Strike rewards combine target approach, facing, movement, tilt, and energy terms, with reward 1 once target tilt exceeds approximately 78°.Source: C Downstream Task Rewards and Settings · p. 20

Pick-and-Place

• Pick-and-place decomposes into pick-up, carry-to, and put-down stages that respectively reward grasping and lifting, transport while maintaining grasp, and stable placement before release.Source: C Downstream Task Rewards and Settings · p. 20

Language-prior guidance

• A gating network predicts part-wise mixture weights over language-derived motion priors and dynamically blends them according to task context.Source: D Language Instruction Sets for… · p. 21, Table 5 · p. 21

Long-horizon composition

• Long-horizon programs use standardized initialization, condition, and termination fields to sequence reusable skill units across heterogeneous behaviors.The two-box sequence reuses pick-up, carry-to, and put-down for the second box after walking to its source table.Source: E Long-Horizon Skill Composition… · p. 21