PaperlayerView exact revision on arXiv

Source-linked AI summary

HA-DiPO: Diffusion Policy Optimization for Heterogeneous Cooperative Transport

Xuebin Li, Jinpeng Xiao, Zong Chen, Pierluigi Arpenti, Bruno Siciliano, Jiebao Sun, Yiqun Li

arXiv:2610.33584v1cs.RO

TL;DR

Heterogeneous robots transporting an unfixed shared payload must coordinate different motion capabilities, while existing approaches leave challenges around flexible policies and decentralized heterogeneous coordination. HA-DiPO separates coordination from robot-specific control and trains local-observation diffusion policies; simulations report improved transport metrics, and experiments demonstrate robot-robot and human-robot transport.

  • Problem

    Heterogeneous robots supporting an unfixed payload must coordinate despite differing motion capabilities, while existing approaches leave open how to learn flexible, robot-specific coordination from local observations.

    Source: §I · p. 1

  • Method

    HA-DiPO separates team coordination from embodiment-specific control and trains independent diffusion policies with centralized team-value guidance, using reward-driven simulator interaction without demonstrations.

    Source: §I · pp. 1–2

  • Results

    HA-DiPO achieves the highest mean survival and velocity-tracking success among evaluated baselines, with tracking success of 0.935 ± 0.074 and planar-velocity RMSE of 0.0476 ± 0.0036 m/s.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Learned policies were deployed on a heterogeneous quadrupedal–wheeled-legged team and with a human partner using only local robot and payload observations.

    Source: §I · p. 2, §VI · p. 7

Abstract

from arXiv · show

Cooperative transport with heterogeneous legged robots requires coordinating different motion capabilities while supporting an unfixed shared payload. We present HA-DiPO (Heterogeneous-Agent Diffusion Policy Optimization), a hierarchical multi-agent reinforcement learning framework under centralized training with decentralized execution (CTDE). The framework decouples team-level coordination from embodiment-specific motion control: at the high level, a centralized critic uses team-value estimates to jointly train independent coordination policies, aligning robot-specific action responses with a shared transport objective; at the low level, embodiment-specific robust motion policies track the coordination commands. The high-level diffusion policies offer more flexible action distributions than single-Gaussian parameterizations, allowing potentially multimodal coordination responses to be represented. The coordination policies learn from scratch through reward-driven simulator interaction without demonstrations. At execution, each robot generates coordination commands using only local robot and payload observations, without partner-state inputs or inter-robot messages. Simulation comparisons show that HA-DiPO achieves higher final mean returns and velocity-tracking success rates, as well as lower velocity errors, than the evaluated baselines. Real-world experiments further demonstrate quadrupedal-wheeled-legged and human-robot cooperative transport.

I. INTRODUCTION

HA-DiPO addresses the challenge of heterogeneous robots coordinating through an unfixed shared payload by separating team coordination from robot-specific locomotion. It combines value-guided diffusion policies with decentralized execution from local observations.

I. INTRODUCTION

  • Unfixed-payload transport couples robot actions through contact, so differences in motion response make identical commands insufficient for consistent payload motion.Existing approaches may depend on predefined formations, dedicated coupling mechanisms, or external motion capture; common single-Gaussian actors also limit multimodal coordination responses.Source: §I · p. 1
  • HA-DiPO separates team-level coordination from embodiment-specific locomotion, using centralized team-value guidance to train independent diffusion actors from simulator rewards without demonstrations.Pretrained robust low-level policies track commands and remain frozen during high-level coordination learning.Source: §I · pp. 1–2
  • Diffusion actors provide greater flexibility than single-Gaussian policies for representing distinct coordination responses, and sequential updates use the latest teammate policies.A variational objective over diffusion paths supports training without explicit action-density evaluation.Source: §I · pp. 1–2
  • At execution, each robot uses local robot and payload observations without partner-state inputs or inter-robot messages; experiments demonstrate heterogeneous robot and human-robot transport.Simulation comparisons report gains over evaluated baselines.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior work addresses transport through model-based control, reinforcement learning, or demonstrations, while heterogeneous and human-robot approaches also use interaction feedback or jointly trained policies.The cited literature includes task-priority optimization, layered control, passive-arm MPC, tether-suspended transport, and carrying learned from demonstrations or interaction.Source: §II · p. 2
  • HA-DiPO brings expressive, robot-specific coordination to loose-contact transport by combining centralized team-value guidance with independent diffusion actors trained from scratch.This approach connects reward-driven policy optimization and expressive diffusion policies with heterogeneous cooperative learning.Source: §II · pp. 2–3

III. DIFFUSION POLICY OPTIMIZATION

HA-DiPO optimizes independent diffusion policies using centralized team values and per-agent path regularizers. Each actor learns a local response to team value under current teammate behavior, with diffusion policies able to represent potentially multimodal responses.

III. DIFFUSION POLICY OPTIMIZATION

  • HA-DiPO extends sequential heterogeneous-agent policy optimization by combining centralized team-value evaluation, per-agent path regularizers, and independent diffusion policies conditioned on local observations.The framework incorporates path-based variational learning into the heterogeneous multi-agent setting.Source: §III · p. 3

A. From team rewards to local response distributions

  • The objective balances team reward and exploration through entropy regularization, with each robot’s temperature controlling its entropy contribution.The local team-value function evaluates a robot’s action under a fixed centralized action-value estimate and current teammate policies.Source: A. From team rewards to local response… · p. 3
  • A robot’s local team value averages centralized action value over teammates’ observations and actions under their current policies, given the robot’s local observation.The resulting value quantifies expected team value for the robot’s selected action in the context of teammate behavior.Source: A. From team rewards to local response… · p. 3
  • The unrestricted entropy-regularized response is a Boltzmann distribution over actions, but its action-value-determined shape may be multimodal and difficult to sample directly.Diffusion actors are used to approximate this response; single-Gaussian actors are unimodal before action squashing.Source: A. From team rewards to local response… · p. 3

B. A neural sampler for bounded actions

The bounded-action sampler uses a diffusion process conditioned on local history, with a reference drift and learned neural correction. It converts the terminal latent sample into a bounded physical action.

B. A neural sampler for bounded actions

  • An Ornstein–Uhlenbeck reference drift supplies mean reversion, while a neural correction shapes a diffusion policy conditioned on local history.The reverse process acts as a sampler for complex target distributions.Source: B. A neural sampler for bounded actions · p. 3
  • The neural network maps the latent sample, sampling time, and local history to a drift correction, allowing multimodal endpoint distributions despite Gaussian transitions.The diffusion process is discretized with Euler–Maruyama using a sequence of sampling times.Source: B. A neural sampler for bounded actions · p. 3
  • The policy transforms only the endpoint latent sample into a physical action using tanh and affine rescaling to the action bounds.Source: B. A neural sampler for bounded actions · p. 3

C. A computable replacement for action entropy

Because diffusion policies lack tractable action densities, HA-DiPO replaces direct action-entropy evaluation with a computable path-based lower bound. The resulting surrogate return is used by both actor and critic.

C. A computable replacement for action entropy

D. Centralized learning with sequential actor updates

A centralized critic is trained with Bellman backups, then actors are updated sequentially against the latest teammate policies. The actor objective uses diffusion paths without explicit action-density evaluation, and the paper states a conditional monotonic-improvement guarantee.

  • The centralized critic uses a one-step Bellman backup with fresh next-step actions and path scores, while Huber loss bounds gradients for large temporal-difference errors.Source: D. Centralized learning with… · p. 4
  • Actors update in a sampled order with the critic frozen, using updated policies for preceding teammates and treating teammate actions as constants.Only the active actor is updated at each step.Source: D. Centralized learning with… · p. 4
  • The actor objective combines Boltzmann response matching with an entropy-bound-gap penalty and avoids explicit action-density and Boltzmann-normalizer evaluation.Source: D. Centralized learning with… · p. 4
  • At execution, each robot uses only its local sampler and frozen low-level controller.Source: D. Centralized learning with… · p. 4
  • With fixed temperatures and reference paths, the surrogate return improves monotonically under exact Markov evaluation and statewise coordinate improvement.The stated result is given in Proposition 2.Source: D. Centralized learning with… · p. 4

IV. HIERARCHICAL CONTROL AND ONBOARD PERCEPTION

The hierarchy separates team coordination from robot-specific motion control, while onboard sensing estimates payload geometry for local feedback. Coordination policies use local observations; low-level controllers track robot-specific motion commands.

A. Hierarchical control

  • Under CTDE, decentralized high-level policies map local robot and payload observations to bounded base-motion commands, which connect coordination to low-level control.The shared team command specifies the desired payload planar velocity and yaw rate.Source: A. Hierarchical control · p. 4
  • Robot-specific low-level policies track commanded planar velocity, yaw rate, base height, and pitch; they are pretrained with dynamics randomization and pushes, then frozen during coordination training.Actor histories exclude partner poses, velocities, actions, and messages.Source: A. Hierarchical control · p. 4

B. Geometry-Constrained Onboard Payload Perception

  • Onboard payload perception uses 3-D LiDAR and IMU measurements to estimate payload center and axis; hardware uses five reflective markers with corresponding observation-model and geometric constraints.Source: B. Geometry-Constrained Onboard… · pp. 4–5

1) Marker geometry and association:

The perception pipeline uses known marker spacing and temporal predictions to associate LiDAR clusters with rod markers, then tracks and constrains their geometry to estimate payload center and axis.

1) Marker geometry and association:

  • For three retroreflective bands spaced by δ, marker positions encode the rod center and unit axis, with outer-marker separation L = 2δ; gated LiDAR returns are clustered and associated using geometry and temporal predictions.Range, region-of-interest, and intensity gates retain mainly band returns before 3-D DBSCAN clustering.Source: 1) Marker geometry and association · p. 5

2) Constrained Kalman filtering:

  • A constant-velocity Kalman filter predicts marker positions and velocities, corrects from detected associated cluster centers, and enforces fixed outer-marker separation L.The resulting midpoint and normalized outer-marker difference give the center and axis, transformed into the robot’s gravity-aligned frame.Source: 2) Constrained Kalman filtering · p. 5

V. EXPERIMENTAL EVALUATION

The evaluation tests HA-DiPO’s transport performance against multi-agent baselines and examines policy representation and heterogeneous deployment. HA-DiPO leads the reported final return, velocity-tracking success, and planar-velocity error comparisons.

V. EXPERIMENTAL EVALUATION

  • The evaluation asks whether HA-DiPO improves transport over MARL baselines, whether diffusion actors better represent distinct high-value responses than Gaussian policies, and whether decentralized behavior transfers to heterogeneous or human partners.Source: §V · p. 5
  • Each diffusion network uses three 256-unit hidden layers, GELU activations, Fourier time embeddings, and eight sampling steps.Source: §V · p. 5

A. Cooperative Transport Performance

  • HA-DiPO achieves the highest final mean return and a 0.0634 ± 0.0019 m/s planar-velocity error, a 37.3% reduction relative to MAPPO; it also leads seven-command tracking success at 0.935±0.074 and mean planar RMSE.Source: A. Cooperative Transport Performance · p. 6
  • The B2–B2W simulation uses seven nonzero commands, matched evaluations, and at least three training seeds with approximately 50 million transitions per seed.Best recorded mean-training-reward checkpoints are evaluated across 448 episodes per seed.Source: A. Cooperative Transport Performance · p. 6, Table I · p. 6
  • Velocity-tracking success requires at least 90% of samples after 2 seconds to meet both planar-velocity error ≤ 0.10 m/s and payload tilt ≤5°.All methods use the same tolerances.Source: A. Cooperative Transport Performance · p. 6

B. Policy Representation Analysis

The analysis compares diffusion and Gaussian policies on a bimodal bandit, then examines whether transport values also favor separated coordination responses. The results show both modes retained by diffusion policies and distinct translation and pivoting maxima in a visited transport state.

B. Policy Representation Analysis

  • The bandit has two separated high-reward regions, making its Boltzmann optimum bimodal; the controlled comparison removes critic-estimation error and partner-policy changes.The Boltzmann target uses temperature α = 0.1.Source: B. Policy Representation Analysis · pp. 6–7
  • 0.83 ± 0.00 for the diffusion actor versus 0.65 ± 0.15 for the Gaussian actor on E[r] + αH; diffusion policies retained both modes across three seeds.The bandit comparison uses a common sample-based physical-action entropy estimate.Source: B. Policy Representation Analysis · p. 7, Figure 5 · p. 7
  • At one visited transport state, translation and pivoting correspond to distinct clipped double-Q maxima, with lower values along the straight segment between them.The forward-velocity sweep fixes all other action coordinates.Source: B. Policy Representation Analysis · p. 7, Figure 5 · p. 7

C. Real-World Experiments

HA-DiPO’s evaluations report strong simulation performance and demonstrate heterogeneous robot and human–robot transport. The experiments also show transfer to a loaded payload configuration and differing command–velocity responses across robot embodiments.

1) Heterogeneous Robot Transport:

  • A B2–B2W team transported an unfixed rod during straight travel and turning, and supported loaded transport without further training.The additional suspended load was an unseen payload configuration; B2 generally moved below command while B2W followed varying commands more closely.Source: 1) Heterogeneous Robot Transport · p. 7

2) Human–Robot Cooperative Transport:

  • In human–robot transport, a B2W paired with a human shares payload support, while a path-integral controller supplies the team command and the robot uses local observations.The robot coordinates with the human through physical interaction with the shared payload.Source: 2) Human–Robot Cooperative Transport · p. 7

VI. CONCLUSION

  • 0.935 ± 0.074 tracking success and 0.0476 ± 0.0036 m/s planar-velocity RMSE; HA-DiPO had the highest mean survival and velocity-tracking success among evaluated baselines.The learned policies were also deployed on a quadrupedal–wheeled-legged team and with a human partner using local robot and payload observations.Source: §VI · p. 7
  • Larger teams, broader payload and environment variations, and more extensive quantitative real-world validation remain future work.Source: §VI · p. 7

APPENDIX A THEORETICAL ANALYSIS

The appendix analyzes the entropy gap associated with path-based response fitting and the conditions under which sequential updates improve a surrogate. It treats temperatures and reference paths as fixed and considers full diffusion-path policies conditioned on Markov states.

A. Entropy gap and response fitting

  • The entropy-gap identity links the conditional log-density ratio to a negative expected KL term and the change of variables to policy entropy.The actor-loss interpretation balances response matching against the conditional-path gap; a finite sampler need not exactly realize the Boltzmann response.Source: A. Entropy gap and response fitting · p. 8

APPENDIX A THEORETICAL ANALYSIS

B. Monotonic improvement of the surrogate

  • Under bounded rewards, fixed positive temperatures and reference paths, and bounded expected absolute path scores, statewise nondecreasing coordinate updates yield the stated return inequality after a completed sweep.The proof chains coordinate inequalities and uses a monotone, γ-contractive surrogate Bellman operator.Source: B. Monotonic improvement of the… · p. 8

Related papers

Loading 2610.33584v1…