PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressively Learning Heterogeneous Skills in a Unified Latent Space

Yue-Yi Zhang, Ming Gong, Linpu He, Wei-Shi Zheng, Zhilin Zhao

arXiv:2608.23258v1cs.CVcs.AIcs.RO

TL;DR

Physics-based character control lacks a unified executable representation for progressively accumulating heterogeneous skills across varied goals and data. HetSkills builds such a representation through a shared latent interface and reusable tracking-based decoder, achieving effective cross-task skill reuse and robust text-to-motion performance. The framework reaches 96.9% success under standard initialization and 81.7% from a neutral pose, while adapting to downstream tasks without task-specific demonstrations or reference motions.

  • Problem

    Physics-based character control lacks a unified executable skill representation that can consolidate heterogeneous goals and support skill accumulation, composition, and transfer.

  • Method

    HetSkills progressively learns heterogeneous skills in a shared latent space, reusing a tracking-trained motion decoder and applying part-wise control, Motion Intuition Distillation, and language-guided adaptation.

  • Results

    HetSkills supports stable heterogeneous-skill reuse, robust text-to-motion generation, and downstream adaptation without task-specific demonstrations or reference motions.

  • Takeaways & Limitations

    A unified latent space can serve as an effective substrate for progressively expanding physics-based character capabilities while preserving natural motion quality.

  • Takeaways & Limitations

    Experiments do not yet fully cover larger-scale skill accumulation, complex long-horizon composition, or open-ended human-object and human-scene interactions.

Abstract

from arXiv · show

We propose HetSkills, a novel framework designed to progressively learn heterogeneous skills within a unified latent space for physics-based character control. The core idea is to treat this latent space as a shared executable interface, enabling seamless integration of skills learned from diverse data sources, supervision forms, and tasks. HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder, which can be reused across tasks without the need for retraining or separate controllers. To prevent the text-to-motion skill from exploiting shortcut pathways instead of learning language semantics, we introduce motion intuition distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions. This enables HetSkills to preserve natural motion while continuously expanding its skill repertoire, making it highly adaptable for long-horizon tasks. Experimental results demonstrate the effectiveness in motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, achieving impressive success rates even under challenging conditions.

1 Introduction

HetSkills addresses the lack of a unified, executable skill representation for physics-based character control by progressively accumulating heterogeneous skills in a shared latent space. It supports skill reuse and compositional control while preserving a coherent motion prior, with robust text-to-motion generation under challenging initializations.

  • Motivation: Physics-based character control lacks a unified executable skill representation that can consolidate heterogeneous behaviors and transfer them efficiently to downstream tasks.Existing methods often rely on task-specific controllers or narrowly curated datasets, limiting reusability and compositionality.
  • Contribution: HetSkills progressively learns motion tracking, text-to-motion generation, motion completion, and downstream task adaptation within one shared latent representation.The latent space acts as a common executable interface for skill acquisition, composition, and reuse.
  • Method: The framework uses part-wise character decomposition so body parts receive specialized yet coordinated control signals within a common latent architecture.This design is intended to improve compositional control and generalization.
  • Method: The tracking skill establishes a shared motion decoder that subsequent tasks can reuse without retraining separate controllers.The tracking foundation provides the motion-control substrate for later skills.
  • Results: 96.9% success under standard initialization and 81.7% from a neutral pose demonstrate robust text-to-motion generation across initialization conditions.The reported results also cover stable heterogeneous-skill accumulation and compositional control.

2 Related Work

Prior work develops physics-based imitation, large-scale motion priors, language-conditioned control, part-wise learning, and downstream adaptation, but existing skill spaces remain less suited to progressively accommodating heterogeneous skills. HetSkills integrates these directions through a unified latent interface.

  • Physics-based Motion Imitation and Skill Learning: Physics-based character control traditionally uses motion-capture data and reinforcement learning to produce physically plausible behaviors.Foundational tracking policies imitate reference motion clips, while adversarial approaches encourage stylistic realism without explicit motion matching.
  • Large-scale Motion Priors and Language-conditioned Control: Large-scale motion priors distill motor skills into universal physics-based latent spaces, allowing diverse downstream tasks to reuse shared motor representations.Later methods combine motion tracking with richer conditioning signals.
  • Part-Wise Motion Learning: Part-wise structure is used to improve controllability, diversity, compositionality, and interaction-rich behavior synthesis.Physics-based methods use it to compose partial motion priors, decouple imitation objectives, or facilitate part-wise planning.
  • HetSkills: HetSkills progressively integrates motion tracking, text-to-motion, motion completion, and downstream adaptation within a unified latent space.Its architecture reuses shared state and motion-decoding modules after tracking training.
  • Limitations of Prior Work: Existing skill spaces are effective for composition under a fixed interface but less suited to progressively accommodating heterogeneous skill types, supervision signals, and conditioning modalities.This limitation motivates a more flexible unified representation.
  • Downstream Task Adaptation with Motion Priors: Freezing pretrained motion priors and training high-level policies in their latent spaces enables downstream tasks to inherit motion naturalness without retraining low-level controllers.Token-based adaptation methods further target complex or contact-rich scenarios.

3 Problem Formulation

The formulation models physics-based character control as a goal-conditioned Markov decision process, but goal signals vary substantially across skills in modality, supervision, and data distribution. HetSkills therefore maps task-specific goals into a shared latent control space whose decoder produces physical actions.

  • MDP Formulation: Physics-based character control is formulated as a goal-conditioned Markov decision process with state s_t, goal g_t, action a_t, transition dynamics, and reward r_t.The policy acts from the current state and goal while maximizing expected discounted return.
  • Action Execution: Actions are target joint rotations converted into torques through proportional-derivative controllers in the physics simulator.This specifies how policy outputs become physical control signals.
  • Heterogeneous Goals: Goal signals differ across motion tracking, text-to-motion generation, motion completion, and downstream adaptation in modality, supervision, and data distribution.They include future poses, language descriptions, sparse temporal constraints, and task-specific observations or language conditions.
  • Unified Control Interface: Each skill predicts a latent z_t from its own goal specification, and a shared decoder maps z_t and s_t to the final physical action.The latent space must incorporate new skills while preserving earlier behaviors and enabling downstream transfer.

4 HetSkills: Progressive Heterogeneous Skill Learning

HetSkills treats a unified latent space as an executable interface for progressively acquiring, composing, and reusing skills learned from heterogeneous sources and supervision. It begins with tracking and builds later goal-conditioned skills on the resulting fixed control substrate.

  • Unified Latent Interface: HetSkills incorporates skills learned from different data sources, supervision forms, and training stages into a common control representation.The unified latent space is explicitly designed for skill acquisition, composition, and reuse.
  • Tracking Foundation: The tracking skill simultaneously constructs the unified latent space and produces the shared motion decoder reused across subsequent stages.It also uses coordinated body partitions with dedicated latent commands within a common architecture.
  • Progressive Skill Learning: Subsequent skills introduce distinct goal-conditioning mechanisms over the fixed control substrate.Text-to-motion uses Motion Intuition Distillation to ground language descriptions into latent motion commands, while motion completion handles sparse or partial observations.

5 ℱ𝑡𝑟𝑐: Tracking Skill

HetSkills first learns a tracking controller with a compact, deterministic latent space and coordinated arm/body branches. Its state encoder and low-level controllers are then frozen as a shared motion decoder for subsequent skills.

  • Architecture: The tracking skill factorizes control into arm and body branches that predict compact latent commands decoded into joint-level target actions.A shared state encoder captures full-body context, while part-wise controllers specialize in the two partitions.
  • Decoder reuse: After training, the shared state encoder and part-wise controllers are frozen and reused as the common motion decoder across later skills.This preserves a common control space without relearning low-level dynamics or requiring an additional distillation step.
  • Inputs: The policy receives proprioception, previous actions, and future reference poses represented by target positions, rotations, and velocity differences.Future goals include K target poses, with relative positions and pose errors expressed in the current root frame.
  • Part-wise architecture: The two-part decomposition specializes upper-limb behaviors while maintaining coordinated whole-body behavior through a shared state representation.The arm branch is motivated by the distinct high-frequency, low-inertia dynamics of gesturing, punching, and object interaction.
  • Latent space: A deterministic, compact latent representation separates motion categories such as locomotion, gesture, and acrobatics without explicit category labels.The learned structure is intended to support later skill reuse and composition.

6 ℱ𝑡2𝑚: Text-to-Motion Skill

The text-to-motion skill maps language and current observations to latent actions while reusing the tracking skill’s frozen low-level controller. Motion Intuition Distillation removes privileged future-motion and initialization shortcuts so language must guide behavior.

  • Skill design: The text-to-motion skill maps natural-language descriptions to arm and body latent actions while preserving the tracking decoder.Only the text-conditioned mapping is learned; the low-level controller remains frozen.
  • Motion Intuition Distillation: Motion Intuition Distillation removes dependence on future motion and standard initialization, requiring inference from language, current state, and imperfect history.Reference State Initialization samples an episode’s initial state from any reference-clip frame with probability p_rsi = 0.7.
  • Shortcut issue: Shortcut pathways can let text-conditioned models bypass meaningful language semantics when future motion or privileged information is accessible.Nearby future frames and matched initial states provide easier predictive routes than semantic inference.
  • Architecture: The skill uses a frozen TMR text encoder and two part-wise Transformer encoders to predict latent actions decoded into physical actions.The inputs include proprioception, the previous action, text embedding, and recent observation history.
  • Shared decoder reuse: Freezing the shared decoder reduces text-to-motion training to latent prediction and allows comparatively small text-annotated data to build on a diverse motion prior.The decoder supplies already learned low-level control rather than jointly learning physical control again.

7 ℱ𝑚𝑜𝑐: Motion Completion Skill

The motion completion skill expresses diverse sparse-observation tasks as partial goal specifications in the shared latent space. It predicts compatible latent actions for full-body behavior from spatially or temporally incomplete conditioning.

  • Unified formulation: Motion completion recovers physically plausible full-body motion from sparse or partial observations across heterogeneous conditioning signals.The framework treats VR tracking, motion in-betweening, and human-scene interaction under a common formulation.
  • Unified formulation: A single skill family maps current proprioception and sparse target observations to latent actions compatible with the shared decoder.This avoids introducing a separate controller for each conditioning type.
  • VR Tracking: VR tracking provides the next-frame head and hand states, requiring the controller to infer the remaining full-body motion.It represents a spatially sparse observation setting and is trained with behavior cloning using the stage-1 tracker as teacher.
  • Motion In-betweening: Motion in-betweening supplies a future full-body pose and time offset, with target frames sampled 5 to 30 frames ahead.The controller synthesizes plausible intermediate motion before resampling another future target.
  • Human-scene Interaction: Human-scene interaction uses future target interaction states and time offsets to model motions involving everyday objects such as chairs and sofas.The SAMP dataset lies outside the tracker’s training-data distribution.

8 ℱ𝑡𝑠𝑘∗: Language-Guided Downstream Adaptation

HetSkills adapts a frozen, language-conditioned motion prior to downstream objectives through a standardized task interface and lightweight, compositional guidance modules. Latent regularization keeps adapted actions within a stable shared control space while preserving natural motion.

  • HetSkills reuses the language-conditioned motion distribution for new objectives without retraining the motion prior or low-level controllers.
  • Standardized Task Description: A standardized task unit represents initialization, execution condition, and termination, allowing heterogeneous skills to be specified, organized, composed, and extended consistently.
  • Task Composition: The shared interface supports independent execution and sequential organization of language-conditioned motion, interaction, and goal-conditioned locomotion skills for long-horizon composition.
  • Compositional Task Guidance: Multiple part-wise language priors are routed according to task context and refined with task-conditioned residuals to represent complex or changing downstream behaviors.
  • Optimization: Downstream adaptation trains lightweight guidance modules with PPO while keeping pretrained priors and decoders fixed.The learned modules include part-wise gating and residual networks, together with a value network.
  • Optimization: Latent magnitude and temporal smoothness penalties counter task-reward-driven drift, keeping adapted actions stable and improving motion naturalness and robustness.

9 Experimental Setup

The experiments evaluate HetSkills across progressive training stages, heterogeneous datasets, motion skills, text-to-motion generation, and demonstration-free downstream tasks. The setup includes tracking, motion completion, semantic and pose robustness, and adaptation to path-following, striking, and pick-and-place objectives.

  • Experiments use Isaac Lab with ProtoMotions, 120 Hz physics simulation, and 30 Hz policy execution, training across four progressive stages on two RTX 5090 GPUs.
  • Progressive training uses heterogeneous data sources without requiring a single annotation format, allowing each stage to use task-appropriate data.
  • Tracking and Motion Completion: Motion tracking uses filtered AMASS sequences, with the cleaned set also reused for VR tracking and motion in-betweening.Filtering removes non-physical artifacts including limb penetration, body floating, and interactions with unmodeled objects.
  • Downstream Tasks: Downstream evaluation covers online path following, target striking with hands or legs, and staged pick-and-place behavior without task-specific demonstrations.The path-follow task uses randomly generated paths through complex terrain, while strike targets are randomly placed.
  • Tracking and Motion Completion: Tracking and motion completion are evaluated with success rate and MPJPE across AMASS training and test splits and sparse spatial or temporal conditions.
  • Text-to-Motion: Text-to-motion evaluation measures pose-level robustness under ground-truth-first-frame and neutral-pose initialization, plus semantic alignment using HumanML3D retrieval.
  • Downstream Task Adaptation: Downstream adaptation is evaluated on the three designed tasks against relevant baselines, with language instructions specified separately.
  • Tracking and Motion Completion: Qualitative tracking examples assess whether the shared decoder reproduces diverse reference motions with stable balance, plausible contacts, and coordinated full-body control.

10 Experimental Results

HetSkills demonstrates that a shared latent space can support heterogeneous motion skills, robust language-conditioned generation, sparse motion completion, and downstream adaptation. The framework also composes these skills into coherent long-horizon behaviors without modifying the shared controller or manually designing low-level transitions.

  • Tracking: HetSkills-1step achieves the highest success rate under fair inference, while duplicating one observed frame provides nearly the same performance as the two-frame variant.The two variants differ because tracking was trained with a two-frame future window; HetSkills-1step duplicates a single frame at inference.
  • Tracking: 42.3 mm to 35.5 mm: increasing part granularity from one to five parts reduces test-set MPJPE, but adds computational overhead and reduces semantic interpretability.The authors therefore adopt a two-part arm-and-body decomposition as a balance between accuracy and efficiency.
  • Motion Completion: 99.9% and 97.8%: HetSkills succeeds on training and test splits for VR-driven motion completion from head-and-hand observations.The lower body is underdetermined by only three upper-body endpoints, so test-set positional errors may not reflect motion quality.
  • Motion Completion: 99.9% training and 100% test success: HetSkills performs motion in-betweening on AMASS, while achieving 96.1% on human-scene interaction.These results support compatibility among skills trained on different motion distributions and conditioning modalities, although success rate does not fully capture plausible intermediate trajectories or naturalness.
  • Text-to-Motion: 96.9% and 81.7%: HetSkills outperforms MaskedMimic and CLoSD under first-frame and neutral-pose initialization for text-to-motion generation.Removing RMI reduces neutral-pose performance, while adding a residual branch beside the text encoder fails under both protocols.
  • Text-to-Motion: HetSkills preserves strong text-motion alignment under neutral-pose initialization, with higher R@N and lower MedR than the baselines.These retrieval metrics complement MPJPE-based success rates by evaluating whether generated motion remains recognizable as the intended textual action.
  • Downstream Task Adaptation: Semantic guidance helps downstream policies preserve natural motion and explore task-relevant latent regions, enabling reliable convergence on coordinated pick-up.Without semantic guidance, the sparse bimanual-lifting region is difficult to discover within the limited sample budget.
  • Downstream Task Adaptation: The MoE module dynamically selects and combines language priors according to task context, including complementary instructions during low-activity phases.This supports smooth transitions and coherent long-horizon behavior across heterogeneous task stages.

11 Limitations and Future Work

The experiments identify scope and robustness limitations, while future work targets broader interaction coverage, stronger contact modeling, better language-motion alignment, and open-ended task planning.

  • Current experiments do not fully cover larger-scale skill accumulation, more complex long-horizon composition, or open-ended human-object and human-scene interactions.
  • Text-driven motion generation still struggles with high-difficulty actions and ambiguous language descriptions.Motion Intuition Distillation improves robustness to different initial states and motion histories, but annotation quality remains limiting.
  • Semantically similar descriptions can correspond to substantially different motions, making fine-grained language-motion alignment difficult.
  • Future work proposes more diverse contact-rich and scene-aware data, contact-aware modules, higher-quality text-motion annotations, broader skills, and real-world robot control.
  • More robust task decomposition, automatic skill selection, failure recovery, and language-model-based high-level planning are proposed for open-ended long-horizon tasks.The current work uses an LLM to help organize high-level instructions into task sequences.

12 Conclusion

HetSkills progressively integrates heterogeneous character-control skills in a unified latent space treated as a shared executable interface. The framework supports natural-language control, skill reuse, and downstream adaptation while preserving natural motion quality.

  • HetSkills extends one shared control substrate with skills from different data sources, supervision forms, and training stages rather than dedicated controllers.
  • The framework integrates motion tracking, text-to-motion generation, motion completion, and downstream task adaptation.
  • Natural language serves as an interface for both motion generation and task guidance.
  • Part-wise latent control, motion intuition distillation, and language-guided latent composition support motion preservation, skill reuse, and adaptation without task-specific demonstrations or motion-prior retraining.

A Implementation Details

Implementation uses progressive training: tracking establishes a frozen shared controller, while later skills learn latent mappings or guidance modules on top of it. Separate configurations cover text, sparse observations, and downstream adaptation.

  • Tracking skill: Tracking uses PPO with a 512-dimensional shared state encoder, final-action Gaussian exploration, a 0.5 m termination threshold, and RSI probability 0.8.
  • Text-to-motion skill: Text-to-motion uses separate arm and body Transformer branches with current-state, language, and six history tokens from a 60-step window.
  • Motion completion skill: Motion completion covers VR tracking, motion in-betweening, and human-scene interaction using sparse-goal conditioning.
  • Downstream adaptation: Downstream adaptation freezes the pretrained motion prior and low-level decoder, optimizing only task-guidance modules with 64-dimensional arm and body residual latents.
  • Progressive training: Later stages train independently on the shared latent space, reusing the frozen low-level controller instead of jointly optimizing one monolithic model.
  • Training cost: Each downstream task can be trained within 5 hours on one RTX 5090 GPU, while instruction branches are batched in parallel.

C Downstream Task Rewards and Settings

Downstream adaptation combines task-specific rewards with energy regularization and language-conditioned latent priors. Standardized skill programs sequence these modules into multi-stage behaviors, including sequential box placement and sitting-to-striking tasks.

  • Reward design: Downstream rewards encourage task completion while regularizing motion with energy penalties to remain physically plausible and compatible with the pretrained prior.
  • Path Follow: Path-following rewards track online-generated horizontal target paths using root position, leg energy, speed, acceleration, and termination constraints.The path uses maximum speed 5 m/s, maximum acceleration 2 m/s2, and terminates beyond 1.5 m horizontal error.
  • Strike: Strike rewards combine target approach, facing, movement, tilt, and energy terms, with reward 1 once target tilt exceeds approximately 78°.
  • Pick-and-Place: Pick-and-place decomposes into pick-up, carry-to, and put-down stages that respectively reward grasping and lifting, transport while maintaining grasp, and stable placement before release.
  • Language-prior guidance: A gating network predicts part-wise mixture weights over language-derived motion priors and dynamically blends them according to task context.
  • Long-horizon composition: Long-horizon programs use standardized initialization, condition, and termination fields to sequence reusable skill units across heterogeneous behaviors.The two-box sequence reuses pick-up, carry-to, and put-down for the second box after walking to its source table.
Loading 2608.23258v1…