Source-linked AI summary
Learning Multi-Humanoid Pickup and Transport via Decentralized Object-Centric Control
Bikram Pandit, Mohitvishnu S. Gadde, Aayam Kumar Shrestha, Alan Fern
TL;DR
Humanoids can carry objects of varying size and weight, but some payloads are too large, heavy, or irregular for one robot. The paper introduces decentralized object-centric control for multi-humanoid manipulation, demonstrating teams of up to ten in simulation and cooperative manipulation on hardware.
Problem
Some large, heavy, or irregular payloads exceed what a single humanoid can manipulate, motivating cooperative transport by robot teams.
Source: §I · p. 1Method
decMHT assigns each humanoid a local attachment region and uses decentralized controllers with local observations to coordinate through the shared object without direct communication.
Source: §I · p. 1Results
A single trained controller supports teams of up to ten humanoids across object geometries and attachment configurations, including handoffs, and the approach transfers to two Digit V3 humanoids on hardware.
Source: §I · p. 1Takeaways & Limitations
The attachment-based abstraction spans single-robot pickup, cooperative transport, and handoff while generalizing across team sizes, attachment layouts, and object geometries.
Source: §I · p. 1The formulation assumes robots share a behavioral role and locally track their attachment frames; distinct roles require extending the interface with robot-specific role information.
Source: §IX · p. 8
Abstract
from arXiv · showhide
We study cooperative multi-humanoid pickup and transport of objects with varying size, weight, and geometry, requiring robot teams of different sizes. Our approach uses decentralized object-centric control, where each humanoid is assigned a local attachment region on the shared object and learns to realize pickup and transport through gripperless bimanual pinching. This attachment-based interface provides a common control abstraction spanning single-robot pickup, cooperative multi-robot transport, and robot-to-robot handover, without per-task redesign. We find that policies trained only on single-robot pickup already transfer nontrivially to cooperative settings, suggesting that this abstraction captures much of the structure needed for coordination. At the same time, explicit multi-robot training further improves performance, showing that shared-object coupling introduces coordination dynamics that are beneficial to learn directly. We validate the approach in simulation across varying team sizes and object geometries, and demonstrate sim-to-real transfer on hardware, where the learned controllers enable real humanoids to perform cooperative manipulation tasks.
I. INTRODUCTION
The paper introduces decMHT, an object-centric interface for decentralized multi-humanoid manipulation, and reports simulation and hardware demonstrations. It positions the approach against prior work limited by team size, embodiment, or modeling assumptions.
I. INTRODUCTION
- decMHT specifies local attachment regions and object motion, while decentralized controllers coordinate pickup, transport, and handoff using local observations without direct communication.The abstraction is intended to generalize across team sizes, attachment layouts, and object geometries.Source: §I · p. 1
- Prior cooperative humanoid transport work is sparse, and the cited symmetric position/force framework relies on analytic robot and object models rather than learned generalization across configurations.That framework demonstrates transport with up to four humanoids in dynamic simulation.Source: §I · p. 1
- A single trained controller supports teams of up to ten humanoids across diverse object geometries and attachment configurations, including handoffs across elevation differences.The approach is also demonstrated on hardware with two Digit V3 humanoids.Source: §I · p. 1
II. RELATED WORK
- Model-based multi-robot transport typically assumes known coupled dynamics and fixed team sizes, whereas decMHT is model-free and generalizes across unseen team sizes and attachment layouts.Related decentralized learning methods cover other embodiments or narrower settings, including fixed contacts, paired wheeled robots, or compliant-partner interaction.Source: §II · p. 2
- Work on animated humanoid characters supports flexible cooperative transport, but does not address robot-realistic whole-body control or hardware transfer.The paper targets physically realistic humanoid models and reports deployment on real hardware.Source: §II · p. 2
III. DECENTRALIZED TRANSPORT FRAMEWORK
The framework gives each robot a local attachment specification and converts a shared object-level motion command into a robot-specific command. Robots independently use this interface to approach, attach, track, or hand off the payload.
III. DECENTRALIZED TRANSPORT FRAMEWORK
- Each robot receives an attachment spec once per episode and an object-centric command at each timestep, then executes a local controller using interface inputs and local observations.The interface avoids requiring precise robot-level pose commands while remaining composable across robots and object geometries.Source: §III · pp. 2–3
- The attachment spec describes approximate contact location rather than a fixed contact pose, providing local geometry for approach, collision avoidance, and flush bimanual contact.It specifies intended hand-contact points and frontal clearance without describing the full payload geometry.Source: §III · p. 3
- The spec includes an attachment-frame pose, symmetric hand-point spacing, and frontal clearance; spacing and clearance are fixed per region, while the frame pose is tracked online.Simulation obtains the frame pose from object state; hardware estimates it using an external sensor.Source: §III · p. 3
- A task-level command sets planar velocity, yaw rate, and object height, and a fixed rigid transformation maps it to each robot's local command frame.The offset is constant for a given attachment assignment.Source: §III · p. 3
- An externally set idle-or-track flag supports pickup, transport, placement, and handover, with the receiving robot activated before the releasing robot is deactivated.Each controller knows only its own mode, and transition timing is specified externally.Source: §III · p. 3
IV. DECENTRALIZED CONTROL ARCHITECTURE
The decentralized architecture gives each robot the same policy, conditioned on its local attachment specification and object-centric command. A high-level manipulation policy coordinates upper-body targets and locomotion commands, while a separate low-level policy controls leg motion.
IV. DECENTRALIZED CONTROL ARCHITECTURE
- Each robot uses the same control policy with its own attachment specification and object-centric command, and executes independently without communication.Handover is coordinated externally through mode flags.Source: §IV · pp. 3–4
- A low-level policy maps velocity, yaw-rate, and torso-height commands to target leg-joint positions tracked by PD controllers.Its inputs include joint positions and velocities, IMU orientation, and angular velocity.Source: §IV · p. 3
- The low-level locomotion network is pretrained separately and remains frozen during high-level policy training.The high-level policy uses a two-layer LSTM with a 128-dimensional hidden state per layer to support history-dependent contact behavior.Source: §IV · p. 3
- The high-level policy coordinates arm motion and generates locomotion targets for approach, contact, lifting, and command tracking.Its outputs include upper-body joint targets and locomotion commands; base yaw need not match the object-command yaw rate.Source: §IV · p. 3
V. TRAINING APPROACH
The high-level policy is trained with IPPO in simulated episodes using randomized scene configurations and a staged curriculum. Training starts with single-robot pickup and transport, then fine-tunes in two-robot episodes; policies are evaluated zero-shot with teams of up to ten robots.
V. TRAINING APPROACH
- Training uses IPPO on simulated episodes with randomized scene configurations and a staged curriculum that increases contact, support, tracking, and locomotion difficulty.Training begins with single-robot pickup and transport, then fine-tunes the policy in two-robot episodes.Source: §V · p. 4
- Although training uses at most two robots, both the single-robot-trained and two-robot-fine-tuned policies are evaluated zero-shot with teams of up to ten robots.Source: §V · p. 4
- The single-robot phase develops basic skills needed by every robot in the decentralized system before cooperative fine-tuning exposes the policy to coupled manipulation.Source: §V · p. 4
A. Initial Scene Generation
Training scenes use randomized cuboid payloads to generate diverse attachment specifications, with external support allowing practice at otherwise infeasible initial heights. Scene randomization also varies attachment and command frames, robot starting pose, payload dimensions, and mass.
A. Initial Scene Generation
- Training payloads are cuboids with randomized dimensions and masses, representing local attachment regions on potentially larger or non-box-shaped objects.Single-robot dimensions range from 0.1 × 0.1 × 0.1 m to 0.7 × 0.5 × 0.7 m; two-robot training raises the minimum dimensions to 0.3 × 0.1 × 0.1 m, and mass ranges from 0.5 to 1.5 kg.Source: A. Initial Scene Generation · p. 4
- Attachment frames are sampled over the robot-facing graspable face, while command frames vary in planar offset and cover all vertical-axis yaw angles.Robots begin with randomized longitudinal, lateral, and yaw offsets relative to their assigned attachment region.Source: A. Initial Scene Generation · p. 4
- External support lets policies practice contact at randomized payload heights, then is removed after contact is detected over two consecutive 0.05 s intervals.Support can be re-enabled for release modes, allowing pickup, release, and handoff transitions within one episode.Source: A. Initial Scene Generation · p. 4
B. Training Curriculum
The curriculum builds single-robot attachment and transport skills before introducing two-robot fine-tuning. Its stages progress from assisted contact to unsupported holding, height and full-command tracking, distant approach, and cooperative episodes.
B. Training Curriculum
- The curriculum first develops single-robot approach, bimanual attachment, lifting, and object-command tracking before adding multi-robot training.Cooperative learning therefore starts from a controller that already manipulates its assigned local attachment region.Source: B. Training Curriculum · p. 4
- Assisted contact formation trains approach and flush bimanual contact with the payload externally supported; unassisted holding then requires support and stabilization under normal physics.Object-command tracking is disabled in both stages.Source: B. Training Curriculum · p. 4
- Height tracking enables payload-height commands while planar velocity and yaw-rate remain zero, followed by full-command tracking with planar velocity and yaw-rate commands.Both stages require maintaining contact or a stable hold.Source: B. Training Curriculum · p. 4
- Distant spawning combines approach locomotion, contact formation, lifting, and transport by initializing the robot farther from its attachment region.Source: B. Training Curriculum · p. 4
- Two-robot fine-tuning exposes the decentralized controller to forces from another robot through the shared payload while preserving its interface.Mode transitions also expose training to cooperative and solo manipulation, release, and both directions of robot-to-robot handoff.Source: B. Training Curriculum · p. 4
C. Reward Design
The reward gates behavior by sampled idle or track mode, while training combines approach, contact, holding, and command-tracking objectives. The policy is trained first with one robot and then fine-tuned with two, with domain randomization used to support robustness and sim-to-real transfer.
C. Reward Design
- Mode-gated rewards let one policy learn distinct idle and track behaviors.Idle rewards encourage zero base velocity, nominal torso height, and rest-pose arm positions.Source: C. Reward Design · p. 4
- Track-mode rewards progress from approaching the assigned attachment frame to making hand contact, holding the object, and tracking commanded motion.For low initial payload heights, the holding term induces squatting, contact, and lifting.Source: C. Reward Design · pp. 4–5
VI. TRAINING DETAILS AND DOMAIN RANDOMIZATION.
- The final policy is initialized from single-robot training and fine-tuned with two robots; it is never trained with three or more robots.Evaluations with three or more robots use the policies without additional fine-tuning.Source: §VI · p. 5
- Training randomizes sensor noise, friction, stiffness, motor delay, sensing rate, and external payload perturbations to improve robustness and support sim-to-real transfer.Attachment-spec observations are updated at a reduced rate to model hardware sensing latency.Source: §VI · p. 5
VII. EXPERIMENTS AND RESULTS
- The evaluation asks how performance scales, generalizes across masses and geometries, changes with multi-robot fine-tuning, compares with baselines, and transfers to hardware.The evaluation is primarily in simulation, with hardware validation.Source: §VII · p. 5
A. Experimental Setup
The experiments evaluate teams of one to ten humanoids on pickup, transport, and handover, using randomized configurations and commanded object motions. In the primary box task, pickup remains reliable as teams grow, while several transport-quality measures generally improve.
A. Experimental Setup
- Teams of one to ten robots perform cooperative pickup, transport, and handover, with 1,000 randomized episodes per configuration.Pickup-and-transport episodes last 8 seconds and handover episodes 12 seconds; commands include planar speeds up to 0.6 m/s, yaw rates up to 0.3 rad/s, and heights from 0.4 to 0.95 m.Source: A. Experimental Setup · p. 5
- The evaluation measures pickup success, linear and angular tracking error, height tracking error, object tilt, and drop rate.Tracking errors are RMS errors against commanded object motion or height; drop rate counts drops or lost attachment contact.Source: A. Experimental Setup · p. 5
B. Scalability Across Team Sizes
- Team-size trials distribute robots around the pallet, adjust object dimensions to keep adjacent attachment points at least 0.8 m apart, and scale payload mass from 1 kg to 10 kg.Trials require approach and pickup before tracking hold-still, forward, lateral, or turning commands.Source: B. Scalability Across Team Sizes · p. 5
- RMS object tilt falls from 8.04° with one robot to 1.21° with ten, while pickup success remains above 97% at every team size.Linear and height tracking errors decrease and drop rates remain low as team size grows.Source: B. Scalability Across Team Sizes · p. 5, Table I · p. 5
C. Object Mass vs. Team Size
Increasing team size expands the payload masses that can be lifted and transported, including zero-shot transfer to ten robots carrying up to 50 kg. Across unseen object geometries, decMHT transfers without per-object retraining, while its comparisons reveal both benefits of multi-robot fine-tuning and geometry-specific transport challenges.
C. Object Mass vs. Team Size
- A policy trained with at most two robots generalizes zero-shot to teams of up to ten carrying payloads as heavy as 50 kg.Increasing team size expands the feasible payload range; tracking and stability generally improve as teams grow.Source: C. Object Mass vs. Team Size · p. 6, Figure 3 · p. 6
D. Generalization to Object Geometry and Baselines
- decMHT achieves high pickup success on unseen geometries and masses without per-object retraining, although the four-robot bed task reaches 85% pickup success.The bed's high mass and thin protruding grasp regions are cited as likely contributors to its difficulty.Source: D. Generalization to Object Geometry… · p. 6
- The log has the lowest velocity-tracking error but relatively high tilt, while the cabinet also has substantial peak tilt; fridge and bed show larger height-tracking errors than lower-profile objects.The text relates log tilt to its rounded geometry, cabinet tilt to its tall, top-heavy shape, and height error to references farther from the robots' nominal height.Source: D. Generalization to Object Geometry… · pp. 6–7
- The single-robot-trained decMHT(1) policy has pickup rates comparable to decMHT on most objects, but the bed result falls from 85.2% to 44.1%.Across objects, decMHT(1) also has higher tracking error, tilt, and drop rate, indicating that single-robot training captures attachment and lifting but not coupled transport dynamics.Source: D. Generalization to Object Geometry… · p. 7
- DecMHT performs comparably to centralized and object-specialized baselines retrained per object, and neither baseline consistently outperforms it across objects and metrics.The centralized baseline was omitted for the four-robot bed task because its joint observation and action space made training impractical.Source: D. Generalization to Object Geometry… · pp. 6–7
E. Cost of Transport
CoT largely plateaus after its initial increase from one to two robots, while decMHT maintains lower multi-robot CoT than decMHT(1). Across handover settings, decMHT matches or improves on decMHT(1).
E. Cost of Transport
- The authors interpret the CoT pattern as an initial coordination cost that does not grow substantially with team size.Despite being fine-tuned only with two robots, decMHT's efficiency gains transfer zero-shot to larger teams.Source: E. Cost of Transport · p. 7
F. Handover Evaluation
- Handover tests cover stationary, moving-object, and different-elevation settings, measuring three transfer stages plus peak height deviation and tilt.In cross-level handoffs, the receiving robot adjusts its stance height.Source: F. Handover Evaluation · p. 7
E. Cost of Transport
- CoT rises sharply from one to two robots, then largely plateaus; decMHT has lower CoT than decMHT(1) for every multi-robot team.The figure compares CoT by team size with payload mass scaled linearly with team size.Source: Figure 4 · p. 7, E. Cost of Transport · p. 7
F. Handover Evaluation
- Across static, dynamic, and cross-level handovers, decMHT matches or improves on decMHT(1), with the largest gains in dynamic handoffs.The table covers handover progress and peak object deviation.Source: Table III · p. 7, F. Handover Evaluation · p. 7
VIII. SIM-TO-REAL TRANSFER
The paper reports sim-to-real transfer to two Digit V3 humanoids, which perform pickup, transport, and handover without additional real-world fine-tuning. It also identifies gripperless pinching, shared behavioral roles, and locally tracked attachment frames as scope boundaries.
VIII. SIM-TO-REAL TRANSFER
- On hardware, two Digit V3 humanoids demonstrate single-robot pickup and transport, cooperative transport, and handover by toggling operating-mode flags without changing the policy, inputs, or training.The trials use a cardboard box and require no additional real-world fine-tuning.Source: §VIII · pp. 7–8
IX. LIMITATIONS
- The current scope is gripperless pinching, shared behavioral roles, and locally tracked attachment frames; grippers and distinct roles remain undemonstrated, and hardware tracking uses external motion capture.Source: §IX · p. 8
X. SUMMARY
- A policy trained with at most two robots generalizes zero-shot to teams of up to ten, diverse payloads, and cross-level handovers, with sim-to-real transfer to two Digit V3 robots.Source: §X · p. 8