PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization

Liang Pan, Zeshi Yang, Zhiyang Dou, Wenjia Wang, Buzhen Huang, Bo Dai, Taku Komura, Jingbo Wang

arXiv:2503.19901v2cs.CV

TL;DR

Existing HSI controllers often specialize in single tasks, limiting unified execution of diverse and composite physical interactions. TokenHSI addresses this with a shared-proprioception transformer policy and lightweight tokenizer-based adaptation, and experiments report improved versatility, adaptability, and extensibility across HSI tasks.

  • Problem

    Existing controllers mainly specialize in individual interaction tasks, limiting support for dynamic manipulation and composite skills such as sitting while carrying an object.

  • Method

    TokenHSI combines a shared proprioception tokenizer with task-specific tokens in a masked, variable-length transformer policy and adapts it using additional tokenizers and lightweight action-head modules.

  • Results

    TokenHSI significantly improves versatility, adaptability, and extensibility across skill composition, object and terrain variation, and long-horizon HSI tasks.

  • Takeaways & Limitations

    A single transformer policy can unify diverse HSI skills while adapting learned behavior to novel tasks and environments without full pretrained-policy fine-tuning.

  • Takeaways & Limitations

    Learning the skills requires tedious trial-and-error reward engineering, and long-horizon task completion remains non-autonomous.

Abstract

from arXiv · show

Synthesizing diverse and physically plausible Human-Scene Interactions (HSI) is pivotal for both computer animation and embodied AI. Despite encouraging progress, current methods mainly focus on developing separate controllers, each specialized for a specific interaction task. This significantly hinders the ability to tackle a wide variety of challenging HSI tasks that require the integration of multiple skills, e.g., sitting down while carrying an object. To address this issue, we present TokenHSI, a single, unified transformer-based policy capable of multi-skill unification and flexible adaptation. The key insight is to model the humanoid proprioception as a separate shared token and combine it with distinct task tokens via a masking mechanism. Such a unified policy enables effective knowledge sharing across skills, thereby facilitating the multi-task training. Moreover, our policy architecture supports variable length inputs, enabling flexible adaptation of learned skills to new scenarios. By training additional task tokenizers, we can not only modify the geometries of interaction targets but also coordinate multiple skills to address complex tasks. The experiments demonstrate that our approach can significantly improve versatility, adaptability, and extensibility in various HSI tasks. Website: https://liangpan99.github.io/TokenHSI/

1. Introduction

TokenHSI targets the limited versatility of task-specific HSI controllers by unifying diverse physical interaction skills in one policy and adapting them to novel scenarios. Its shared proprioception token, task-token masking, and lightweight adaptation support multi-skill learning and complex tasks.

  • Motivation: Existing unified controllers mainly address static interactions, limiting dynamic manipulation and composite tasks such as sitting while carrying an object.This leaves long-term manipulation and synergies among diverse skills insufficiently supported.
  • Approach: TokenHSI uses a single transformer policy with a shared proprioception token combined with task-specific tokens through masking.The design is trained on following, sitting, climbing, and carrying skills.
  • Approach: Variable-length inputs let the policy adapt to new tasks and environments by training additional task tokenizers and action-head adapters instead of fine-tuning all parameters.Prior task tokenizers can be reused when adapting to new contexts.
  • Evaluation: Experiments cover skill composition, object and terrain variation, and long-horizon completion, with TokenHSI significantly outperforming existing methods on these benchmarks.The evaluation emphasizes versatility, adaptability, and extensibility across challenging HSI settings.
  • Contributions: The paper contributes a unified transformer controller, efficient adaptation without full fine-tuning, and a dedicated proprioception tokenizer.These contributions are intended to support both multi-task training and policy adaptation.

2. Related Work

Prior HSI work spans data-driven motion generation and physics-based control, but existing approaches face trade-offs between motion quality, physical plausibility, and skill versatility. Recent unified controllers broaden skill coverage, yet the literature still motivates more general physical interaction control.

  • Data-driven generation: Data-driven methods use large-scale datasets and generative motion models to synthesize high-quality human-scene interaction motions.These approaches often overlook physical plausibility.
  • Physics-based control: Physics-based methods include hand-crafted grasping controllers and trajectory optimization for increasingly complex human-object interaction skills.Trajectory optimization is presented as a versatile approach for complex skills.
  • Unified controllers: Recent unified controllers support broader skill sets through physics-aware imitation, motion manifolds, and additional task-specific controllers.These methods aim to make learned skills more flexible and reusable.

3. Methodology

TokenHSI uses a variable-length transformer policy that separates shared proprioception from task observations, enabling unified skill learning through masked task selection. After training, it adapts to new tasks by reusing tokenizers and adding lightweight task-specific modules.

  • Policy design: The policy uses variable-length inputs so one transformer can incorporate an arbitrary number of task observations.This architecture is designed for both multi-task training and generalization to novel settings.
  • Foundational skill learning: Separate task tokenizers standardize task observations, while a dedicated proprioception tokenizer processes the character state across tasks.The shared transformer and proprioception tokenizer encourage motor knowledge sharing.
  • Policy adaptation: During adaptation, previously learned tokenizers are reused, new task tokenizers are added when needed, and zero-initialized adapters are inserted into the action head.The pretrained proprioception tokenizer remains frozen and supports varied character states.
  • Physics-based control: The control problem is formulated as goal-conditioned reinforcement learning with style and task rewards and trained using PPO.The state combines character and goal states, and PD controllers produce joint target rotations in a 32-dimensional action space.
  • Policy architecture: The architecture contains multiple tokenizers, a transformer encoder, a learnable output embedding, and an MLP action head.The transformer fuses tokenized observations through self-attention before action prediction.
  • Foundational skills: Foundational training covers following, sitting, climbing, and carrying, with task-specific observations for paths, target objects, and box transport.Training samples tasks according to a predefined distribution, with 10% following and 30% for each other task.
  • Task selection: The masking mechanism activates the selected task token while retaining shared character-state and embedding information in attention computations.An active task label suppresses features from non-target tasks.
  • Policy adaptation: Adaptation freezes core pretrained components, adds action-head adapters, trains a new motion discriminator, and introduces or fine-tunes task tokenizers for new scenarios.This supports skill composition, object-shape variation, and terrain-shape variation without full-policy retraining.

4. Experiments

Experiments evaluate TokenHSI on foundational skills, skill composition, object and terrain variation, and long-horizon completion. Across these settings, the reported results emphasize shared-token knowledge transfer, efficient adaptation, and robust execution in increasingly challenging tasks.

  • Experimental scope: TokenHSI is evaluated on foundational skill learning, adaptation to composite and varied-shape tasks, and long-horizon completion.The experiments include skill composition, object-shape variation, terrain-shape variation, and complex-scene task sequences.
  • Foundational skill learning: Removing the shared proprioception tokenizer causes a general success-rate decrease across all tasks, supporting its role in cross-task knowledge sharing.In the Ours (w/o Tprop) variant, proprioception is added separately to each task tokenizer instead.
  • Skill composition: 99.2% success on Climb + Carry shows TokenHSI maintains high performance on the most challenging reported skill-composition task.Baselines degrade to 26.8% for Scratch, 68.3% for CML, and 51.3% for CML (dual) as task difficulty increases.
  • Object shape variation: TokenHSI surpasses all baselines on object-shape variation and retains efficiency and stability advantages in most learning-curve comparisons.Full fine-tuning is more efficient on Chair but overwrites all parameters, whereas TokenHSI targets skill-preserving adaptation and significantly outperforms AdaptNet.
  • Terrain shape variation: On terrain-shape variation, TokenHSI maintains an efficiency advantage and outperforms the baselines while adapting to height-map observations through a new task tokenizer.The evaluation covers stairs, obstacles, and flat ground, and compares TokenHSI with policies trained from scratch and AdaptNet.
  • Long-horizon task completion: After adaptation, TokenHSI completes a long-horizon sequence with fluent transitions and correctly places the box for the subsequent climbing skill.The approach avoids manually designing transition states and jointly fine-tuning multiple policies in the described task.

5. Discussion and Limitations

TokenHSI unifies diverse HSI skills in one transformer network and adapts them to novel tasks and environments. Its main limitations are reward-engineering cost and non-autonomous long-horizon completion.

  • Discussion: TokenHSI learns various HSI skills within a single transformer network and generalizes them to novel tasks and environments through policy adaptation.The authors report improvements in versatility, adaptability, and extensibility in HSI.
  • Limitations: Reward-function engineering remains costly because learning these skills requires tedious trial-and-error processes.The authors identify this as a general limitation of goal-oriented reinforcement learning.
  • Limitations: Long-horizon task completion remains non-autonomous, leaving realistic complex tasks without human guidance as an open problem.The limitation concerns completing complex, long-term tasks in realistic environments.

Supplementary Material

The supplementary material describes a custom simulated character model tailored to SMPL-based motion retargeting and simulation needs. Separate foot collision shapes are used for stairs because meshed feet produce inaccurate contacts.

  • Character Model Creation: The model adjusts lower-body joint positions, replaces box feet with realistic meshes, and upgrades knees to 3-DoF spherical joints.These changes address body-model alignment, foot geometry, and knee articulation.
  • Character Model Creation: The custom model is motivated by incompatibility between SMPL parameters and AMP’s kinematic structure, which otherwise causes unnatural lower-body retargeting.The improved model is designed to support more natural motion transfer.
  • Character Model Creation: For stairs tasks, foot collision shapes revert to rectangular boxes because meshed feet have inaccurate terrain contact in IsaacGym.The box geometry is more simulation-friendly for stair terrain.
  • Character State and Action: Proprioception describes the simulated character state, while actions specify target joint rotations for PD controllers.The state includes rigid-body positions, rotations, linear velocities, and angular velocities; the action space is 32-dimensional.
  • Character Model Creation: Figure A compares AMP’s original model with improved versions for flat-ground and stairs-terrain tasks.The flat-ground and terrain variants differ in their intended task settings.

B. Tasks

The task suite covers foundational skills, skill composition, object and terrain variation, and long-horizon completion. Training uses manually constructed motion references, diverse object datasets, and task-specific early termination conditions.

  • Task Overview: The implementation covers 12 tasks spanning four foundational HSI tasks, three skill-composition tasks, four shape-variation tasks, and one long-horizon task.Task-specific observations and reward functions supplement common settings.
  • Reference Motion Dataset: The reference-motion dataset contains behavior categories associated with the four foundational HSI tasks and is divided into five subsets.The subsets support realistic motion across the task suite.
  • Reference Motion Dataset: The Loco subset provides 12 AMASS sequences of standing, walking, and turning, and is used for every task because each includes walking.These motions cover basic flat-ground locomotion.
  • Task Overview: Table A summarizes each task’s tokens, reference datasets, episode length, and early termination settings.The listed termination conditions include character fall, object fall, path distance, and interaction early termination.
  • Reference Motion Dataset: The Stair, Climb, Carry, and Sit subsets provide task-specific references for stairs, platform climbing, object carrying, and sitting.Carry references combine OMOMO and AMASS sequences, while Climb references come from AMASS.
  • Skill Composition: Composite skills are learned without post-processing Carry and Sit motions into explicit sitting-while-carrying reference sequences.The policy instead learns composition through task guidance.
  • Object Dataset: Object generalization is evaluated with separate training and testing datasets built from 3D-Front models and procedurally generated boxes.The datasets are designed to test performance on diverse unseen objects.
  • Early Termination Condition: Early termination prevents harmful negative samples and adds object-fall, path-distance, and interaction-specific conditions alongside humanoid fall detection.Object height, trajectory deviation, and static interaction completion determine additional termination events.

B.2. Foundational HSI Tasks

The foundational tasks define trajectory following and sitting through geometric target observations and distance-based objectives. Path following uses future trajectory waypoints, while sitting targets a position above the chair seat.

  • Path Following: Path following requires the character to move along a procedurally generated target 2D trajectory τ.The trajectory is represented as a sequence of sampled waypoints.
  • Path Following: At each 0.1-second timestep, the policy observes 10 future waypoints covering the next 1.0 second.Waypoints are obtained by linear interpolation from the trajectory.
  • Path Following: The path-following objective measures the distance between the character’s current 2D root position and the target trajectory.The task uses the character root and trajectory waypoints to evaluate alignment.
  • Sitting: The sitting target is a 3D position 10 cm above the center of the chair seat’s top surface.The observation also includes the interacting object’s pose, facing direction, and bounding-box corners.
  • Sitting: The sitting policy minimizes the distance between the character’s 3D root position and the target sitting position.Its reward combines near- and far-distance terms with weights 0.7 and 0.3.
  • Sitting: The sitting formulation uses the character root’s 3D position and 2D velocity together with the object’s 2D position and directional information.The directional term uses a horizontal unit vector and vector dot products.

B.2.3. Climbing

The climbing task directs the character to stand on a target object at a specified 3D position. Its observations encode target geometry, while rewards encourage root placement and foot lifting.

  • Task definition: The character must stand on a given object with its root joint at a target 3D climbing position.The target is placed 94 cm above the center of the object's top surface.
  • Task observation: The climbing observation includes the target root position and eight 3D bounding-box corner points.The observation has dimension R27.
  • Task reward: The task reward minimizes the 3D distance between the character's root and its target location.The reward combines near, far, and foot-related terms.
  • Task reward: The foot reward encourages lifting the character's feet when it is close enough to the target object.The authors report that this term is crucial for successful climbing-task training.

B.3. Downstream HSI Tasks

Downstream tasks adapt foundational skills through tokenizer reuse, new observations, and composite rewards. The designs cover skill composition, shape variation, and long-horizon completion.

  • Skill Composition: Composite tasks reuse and freeze relevant foundational task tokenizers while constructing combined task observations and rewards.Examples include sitting, climbing, or following while carrying a box.
  • Skill Composition: The Follow + Carry task combines following states with revised carrying states, excluding the box target location because carrying is no longer primary.Its combined observation has dimension R2×10+39.
  • Object/Terrain Shape Variation: For object shape variation, the pre-trained box-carrying tokenizer is fine-tuned while its task observation and reward are reused.Terrain variation adds a tokenizer for a 2×2 m^2 height map sampled at 1024 sensor points.
  • Long-horizon Task Completion: The long-horizon task sequences four foundational skills and activates the next sub-task reward after each preceding step is completed.A new height-map observation uses 625 sensor points over a 1×1 m^2 area.

C. Implementation Details of CML

CML uses a fixed low-level meta policy and a trainable high-level policy for composing skills. The dual variant extends this arrangement to combine two meta policies through joint-wise weighting.

  • CML: CML is hierarchical: a fixed meta policy acts as the low-level controller, while a new trainable policy acts as the high-level controller.The high-level policy observes proprioception and a new task observation; the meta policy observes proprioception and a base task observation.
  • CML: For Climb + Carry, the specialist climbing policy serves as the meta policy and a new policy generates additional actions.The high-level policy is trained to cooperate with the low-level policy for composite-task learning.
  • CML (dual): CML (dual) adds a second meta policy and has the high-level policy output an additional set of weights.These joint-wise weights are applied to the two sets of meta actions and passed through sigmoid activations into [0, 1].
  • CML (dual): The resulting action composition combines the two meta-policy action sets using the generated joint-wise weights.This operation is described as post-composition in the main paper.

D. Quantitative Evaluation on Long-horizon Task Completion

The long-horizon benchmark evaluates four sequential sub-tasks using completed-sub-task count across 512 trials. TokenHSI achieves the highest score and retains an efficiency advantage over the baselines.

  • Experimental Setup: The long task sequences following a trajectory, carrying a box, climbing onto it, and sitting on a chair on a high platform.A finite-state machine monitors execution through sub-goals and spatial relationships.
  • Experimental Setup: The completed sub-task count is the evaluation metric, with a maximum value of 4, measured over 512 trials.Sub-goals are procedurally generated for each stage.
  • Baselines: The comparison includes Scratch, which learns the whole task from scratch, and Finetune, which iteratively fine-tunes multiple specialist policies.Both Scratch and TokenHSI can observe a height map.
  • Quantitative Results: 3.79 ± 0.14 completed sub-tasks is achieved by TokenHSI, compared with 0.82 ± 0.06 for Scratch and 1.86 ± 0.02 for Finetune.The reported convergence curves also show TokenHSI maintaining an efficiency advantage.
  • Extensibility: The qualitative extension combines a newly learned lifting skill with previously learned sitting and path-following skills.The additional skills include pushing down a large object and walking while lifting a box with two hands.
Loading 2503.19901v2…