PaperlayerView exact revision on arXiv

Source-linked AI summary

From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction

Zekai Deng, Kangyi Chen, Ye Shi, Jingya Wang

arXiv:2610.11722v1cs.RO

TL;DR

Multi-human HOI requires coordination around shared objects, while task-specific adaptation entangles motor execution and coordination. This paper distills single-human teachers into a reusable object-oriented skill with high-level proxy-motion planning, and reports composable learning across interaction types, object geometries, and team sizes.

  • Problem

    Multi-human HOI requires coordinating contact locations, motion directions, and force application around a shared object, while task-specific adaptation entangles motor skill and coordination and limits reuse.

    Source: §1 · p. 1

  • Method

    The framework distills teacher rollouts into a frozen low-level Object-oriented Motion Skill and has high-level policies generate region-wise object-oriented actions for downstream tasks.

    Source: §1 · p. 2

  • Results

    The skill supports robust proxy-motion execution and composable policy learning across interaction types, object geometries, and team sizes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    High-level policies can compose proxy-level object motions while reusing the same low-level skill across diverse single- and multi-human HOI tasks.

    Source: §3.1 · p. 4

    The framework does not address composition across heterogeneous skill types or long-horizon structures such as multi-stage task decomposition and dynamic role reassignment.

    Source: §5 · p. 8

Abstract

from arXiv · show

Physics-based human-object interaction has achieved robust single-agent manipulation skills, yet extending them to multi-agent cooperative tasks remains challenging. Existing approaches typically adapt interaction policies through task-specific fine-tuning, which entangles low-level contact-rich execution with high-level coordination and limits reuse across object geometries, interaction types, and team sizes. We propose a hierarchical framework that converts a single-agent HOI policy into a reusable Object-oriented Motion Skill. Specifically, we reinterpret teacher rollouts as object-oriented action supervision by extracting short-horizon object-proxy motions from executed trajectories, and distill task-specific teachers into a low-level skill operating in an Object-oriented Action Space. For downstream tasks, the distilled skill is frozen as a reusable executor, while a high-level policy coordinates multiple agents by generating region-wise object-oriented actions conditioned on the shared object, task goal, agent states, and local manipulation regions. This formulation shifts multi-agent HOI learning from direct contact-rich full-body control to compact object-level proxy-motion coordination. Experiments on diverse HOI tasks show that the distilled Object-oriented Motion Skill supports robust proxy-motion execution and enables composable policy learning across different interaction types, object geometries, and team sizes.

1 Introduction

Multi-human HOI requires coordinating contacts, motion directions, and forces around a shared object, while task-specific fine-tuning entangles that coordination with low-level motor skill. The paper proposes a reusable object-oriented skill and high-level proxy-motion planning, reporting composable learning across interaction types, object geometries, and team sizes.

1 Introduction

  • Multi-human HOI requires coordinating contact locations, motion directions, and force application around the same manipulated object.This creates a coordination challenge beyond single-human HOI.Source: §1 · p. 1
  • Task-specific transfer can solve a cooperative task but entangles low-level motor skill with high-level coordination, limiting reuse and transfer.The stated boundaries include new object geometries and cooperative configurations.Source: §1 · p. 1
  • The Object-oriented Action Space uses short-horizon object-proxy motions so the low-level policy learns their realization through contact selection, balance, and full-body coordination.The interface is intended not to be tied to a specific task goal or over-constrain humanoid motion.Source: §1 · p. 2
  • The framework distills single-human HOI teachers into a reusable Object-oriented Motion Skill and uses high-level policies to coordinate humans through short-horizon object-proxy motions.The low-level skill executes contact-rich motions; downstream policies reason in the Object-oriented Action Space.Source: §1 · p. 2
  • The framework supports robust proxy-motion execution and composable learning across interaction types, object geometries, and team sizes.The contribution is validated on cooperative HOI tasks.Source: §1 · p. 2

2 Related Work

Related work spans kinematic motion synthesis, physics-based character control, and cooperative interaction, but these approaches leave gaps for physically valid, reusable humanoid object manipulation. The paper contrasts its reusable low-level proxy-motion skill with methods that remain tied to single-human settings or task-specific cooperative coordination.

Human-Object Interaction Motion Synthesis and Physics-Based Control

  • Kinematic motion synthesis covers human-scene and human-object interaction, but often lacks physical validity for long-horizon contact-rich interactions.Such interactions require sustained contacts, force exchange, and object dynamics.Source: §2 · pp. 2–3
  • Physics-based character control supports physically plausible full-body behavior and contact-rich HOI, but control abstractions are usually tied to references, task labels, or single-human settings.The cited extensions include object manipulation, scene interaction, sports skills, and whole-body HOI control.Source: §2 · p. 3

Multi-Character Interaction and Cooperative Object Manipulation

  • Multi-character methods improve physical plausibility but largely target human-human interaction, sports, reaction, or collision avoidance rather than cooperative object manipulation.Kinematic datasets and animation methods enable plausible interactions without guaranteeing valid contacts, forces, or shared object dynamics.Source: §2 · p. 3
  • Robot payload transport and carrying demonstrate object-level coordination, while CooHOI still couples motor execution with task-specific multi-human coordination.The robot systems typically rely on quadrupedal morphology, engineered interfaces, or task-specific control designs.Source: §2 · p. 3

3 Method

The method separates contact-rich humanoid execution from task-level coordination: a reusable low-level skill realizes proxy motions, while high-level policies plan region-wise motions for downstream tasks. It learns the skill from relabeled single-human teacher rollouts, post-trains it on feasible proxy motions, then freezes it for reuse.

3.1 Low-Level Policy: Generalizable Object-oriented Motion Skill

  • The Object-oriented Action Space specifies short-horizon object-anchor displacements, giving the skill dense motion guidance while preserving freedom in contact selection, balance, and locomotion.The target motion is transformed into the receiving humanoid’s local frame; the object proxy supplies geometric and dynamic context for execution.Source: §3.1 · p. 4
  • Teacher rollouts are relabeled as future object-proxy-motion supervision, converting task-specific single-human policies into a reusable low-level executor.Teacher actions are queried online for behavior-cloning supervision rather than stored in the rollout buffer.Source: §3.1 · pp. 4–5
  • During distillation, mixed teacher-student rollouts expose the student to feasible interaction states, with PPO and online behavior cloning conditioned on future proxy motion.Teacher action replacement and the behavior-cloning weight are gradually annealed to zero.Source: §3.1 · p. 5
  • Random feasible proxy motions are used for post-training to broaden the skill beyond teacher-rollout trajectories before it is frozen for downstream use.Post-training removes teacher action replacement and behavior-cloning loss and optimizes proxy tracking.Source: §3.1 · p. 5

3 Method

  • A frozen Object-oriented Motion Skill executes local proxy motions planned by task-specific high-level policies, forming a shared hierarchy for single- and multi-human HOI.Downstream learning trains the high-level policy while reusing the low-level skill.Source: §3 · p. 3, §3.2 · p. 6

3.2 High-Level Policy: Composable Cooperative Planning for Shared Object Manipulation

  • A geometric adapter represents local regions in the skill’s cuboid-proxy format, allowing the same skill to span object regions, geometries, interaction types, and team sizes without architectural changes.Local proxies can correspond to endpoints, sides, or corners of larger objects.Source: §3.2 · p. 6
  • For each humanoid, the high-level policy plans one near-future action for a local proxy using object-centric agent state and the task goal.Manipulation regions are assigned by object geometry and interaction role; the shared skill handles contact-rich execution.Source: §3.2 · p. 6

4 Experiments

The experiments test whether a frozen Object-oriented Motion Skill transfers across compositional HOI tasks and whether it executes teacher-derived and random proxy motions. The reported results support transfer across object geometries, interaction modes, and team sizes, while ablations distinguish the roles of teacher-guided distillation and random-motion post-training.

4.1 Evaluation on Compositional HOI Tasks

  • For downstream tasks, the low-level skill remains frozen while task-specific high-level policies generate local-proxy actions; the evaluation spans multi-human and single-human manipulation settings.Reported measures include success rate and precision for downstream tasks, and proxy-position, velocity, acceleration, target-reaching, and trajectory-following measures for skill learning.Source: §4.1 · p. 6, §4.2 · p. 7
  • CooHOI adapts a full interaction policy for supported carrying tasks, while adapted InterPhys and SkillMimic share the object-oriented target interface and tracking objective.CooHOI is not evaluated on pushing because that task is outside its released carrying-focused setting.Source: §4.1 · p. 7, §4.2 · pp. 7–8
  • The hierarchy transfers across object geometries, interaction modes, and team sizes, including multi-human carrying and pushing with the same frozen skill.Single-human carrying and large-box pushing additionally test geometric transfer.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Evaluation on Object-oriented Motion Skill

  • The skill achieves the best tracking performance on teacher-rollout targets and maintains higher target-reaching and trajectory success rates than adapted baselines on random proxy-motion targets.Table 2 reports tracking under teacher-rollout and procedurally generated random proxy-motion targets.Source: §4.2 · p. 8, Table 2 · p. 8
  • Removing online distillation causes a clear performance drop, especially on random proxy-motion targets, where tracking rewards alone are insufficient to bootstrap contact-rich carrying.Teacher-guided rollout and online behavior cloning provide a feasible interaction prior for stable grasping, balance, and locomotion.Source: §4.2 · p. 8
  • Removing random-motion RL post-training preserves strong teacher-rollout performance but substantially weakens generalization to random proxy-motion targets.Teacher rollouts cover a limited range of executable motions; random-motion post-training expands that range for downstream learning.Source: §4.2 · p. 8

5 Limitations

The framework does not address composition across heterogeneous skill types or long-horizon temporal task structures.

5 Limitations

  • The framework does not address heterogeneous skill-type composition or long-horizon structures such as multi-stage task decomposition and dynamic role reassignment.Skill-level composition with higher-level planning is identified as future work.Source: §5 · p. 8

6 Conclusion

The framework converts task-specific single-human interaction policies into a reusable Object-oriented Motion Skill by relabeling teacher rollouts with short-horizon object-proxy motions. The frozen skill realizes region-wise object-oriented actions through contact-rich full-body control, supporting transfer across object geometries, interaction types, and team sizes.

6 Conclusion

  • The method relabels teacher rollouts with short-horizon object-proxy motions to learn a reusable Object-oriented Motion Skill instead of directly fine-tuning single-human policies into task-specific cooperative controllers.The skill operates under the Object-oriented Action Space and is frozen for downstream use.Source: §6 · p. 8
  • The frozen skill realizes region-wise object-oriented actions through contact-rich full-body control and supports transfer across object geometries, interaction types, and team sizes.Source: §6 · pp. 8–9

A.1 Training and Implementation Details

The low-level skill is distilled from single-human teacher rollouts and post-trained on generated proxy motions, then frozen while high-level policies learn multi-agent coordination. The implementation uses shared simulated humanoids, proxy-based task interfaces, and task-specific rewards.

Character Model and Optimization

  • The humanoid uses 28 actuated degrees of freedom controlled by target joint configurations, with a 223D proprioceptive state; training uses 4096 parallel environments at 30Hz.The low-level skill trains for 15000 distillation epochs and 5000 random-target post-training epochs; high-level training runs 10000, 20000, and 40000 epochs for one-, two-, and four-agent tasks.Source: §A.1 · pp. 15, 19, 20

Low-Level Skill Training

  • Teacher-guided distillation relabels successful rollouts with future proxy motions, anneals teacher-action replacement and behavior-cloning guidance, then trains the student toward autonomous execution.The student uses object-proxy tracking and auxiliary human-motion tracking rewards during distillation.Source: §A.1 · p. 15
  • Random-motion post-training removes teacher guidance and human tracking, exposing the skill to feasible proxy targets beyond teacher rollouts before it is frozen for downstream learning.Only the object-proxy tracking reward remains in this stage.Source: §A.1 · pp. 15–16

Downstream High-Level Policy and Rewards

  • The high-level policy receives shared-object state, task goal, and per-agent inputs, then outputs region-wise proxy-motion conditions transformed into each humanoid’s local frame for execution.Regions are assigned by object geometry and role, including endpoints, sides, corners, or pushing surfaces.Source: §A.1 · p. 17
  • Downstream training freezes the low-level skill and optimizes only the high-level policy; the single-human teacher is independently trained with the object-transport objective.The teacher uses the whole box as its assigned proxy region.Source: §A.1 · p. 17
  • The shared-object reward combines agent holding with transport guidance for direction, velocity, distance, and final placement, with the put-down term active near the target.The holding reward encourages attachment to assigned local proxy regions; the transport terms include dense direction, velocity, and distance guidance.Source: §A.1 · pp. 17–19

Failure mode compared with CooHOI

  • Compared with CooHOI, explicit short-horizon local trajectories and a robustly post-trained skill are less prone to deadlock after failed lifting attempts.The described failure leaves agents near a static object without effective corrective actions.Source: §A.1 · p. 20

Related papers

Loading 2610.11722v1…