PaperlayerView exact revision on arXiv

Source-linked AI summary

DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication

Hanchu Zhou, Dechen Gao, Hang Wang, Brendan Lynch, Boqi Zhao, Qiyao Ma, Raman Goyal, Junshan Zhang

arXiv:2610.02161v1cs.ROcs.AI

TL;DR

Multi-robot coordination requires long-horizon teamwork alongside reliable low-level execution, but prior progress and evaluation have largely centered on single robots or limited settings. DuoMind combines per-robot VLM orchestration, VLA execution, and semantic messaging, and the paper reports improved task execution on RoboPoly and RoboTwin. Its evaluation focuses on two-robot settings.

  • Problem

    Most robot-model progress has focused on single robots, while multi-robot systems need long-horizon coordination and benchmarks provide limited systematic evaluation of that capability.

    Source: §1 · p. 2

  • Method

    DuoMind gives each robot a VLM orchestrator that reasons over instructions, local observations, and peer messages, then sends low-level instructions to a VLA and semantic messages to peers.

    Source: §1 · p. 2

  • Results

    Experiments on RoboPoly and RoboTwin demonstrate improved multi-robot task execution, and ablations highlight explicit information sharing under partial observability.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The framework is designed to use existing pretrained VLM and VLA models with limited adaptation for multi-robot coordination.

    Source: §1 · p. 2

    The evaluation focuses on two-agent settings; coordination at larger team scales is left for future study.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Vision-language models (VLMs) and vision-language-action models (VLAs) have recently driven rapid progress in general-purpose robots, yet most progress has focused on single-robot settings. Extending these capabilities to multi-robot systems remains challenging because robots must coordinate long-horizon behaviors while maintaining reliable, fine-grained execution. We introduce DuoMind, a distributed hierarchical framework for multi-robot coordination through semantic communication. Each robot uses a VLA-based action model for low-level execution and a VLM-based orchestrator for high-level reasoning and inter-agent coordination. At each planning step, the orchestrator at each robot reasons over the task instruction, local observations, and messages received from other robots. It then generates low-level instructions for the action model and semantic messages for peer robots. This architecture exploits the complementary strengths of pretrained models by combining the semantic reasoning capabilities of VLMs with the precise action-generation capabilities of VLAs. To address the scarcity of benchmarks for multi-robot coordination, we further develop RoboPoly, a benchmark comprising long-horizon manipulation tasks that require coordinated, closed-loop execution under distributed control. Experiments on RoboPoly and RoboTwin demonstrate that DuoMind improves multi-robot task performance, while ablation studies confirm the contributions of hierarchical orchestration and semantic communication. More details are available on our project page.

1 INTRODUCTION

The introduction identifies a gap between high-level multi-agent reasoning and precise physical execution, alongside limited benchmarks for evaluating multi-robot coordination. It introduces DuoMind and RoboPoly to address these challenges and reports positive results across cooperative manipulation tasks.

1 INTRODUCTION

  • Multi-robot manipulation demands that robots form subgoals and adapt as the shared workspace changes, while coordinating team behavior with precise low-level execution.Distributed execution is desirable in practical settings because decisions rely primarily on each robot’s own observations and can be more scalable as robot numbers grow.Source: §1 · p. 2
  • The paper situates this challenge between methods tested in simpler, shorter-horizon settings and language-model systems whose low-level control often relies on predefined primitives or rule-based skills.Existing benchmarks also provide limited support for systematic evaluation of multi-robot coordination.Source: §1 · p. 2
  • DuoMind pairs a VLM orchestrator for high-level reasoning with a VLA action model for low-level control, generating both execution instructions and semantic messages for peers.The messages communicate an agent’s intention, subgoal, and task belief under partial observability.Source: §1 · p. 2
  • RoboPoly targets long-horizon multi-robot coordination with closed-loop physical execution, addressing limited support for systematic benchmark evaluation.The benchmark is designed to test coordination rather than high-level planning alone.Source: §1 · p. 2
  • Experiments on RoboPoly and RoboTwin demonstrate DuoMind’s effectiveness, while ablations quantify contributions from hierarchical orchestration and inter-agent communication.The introduction reports these findings across challenging cooperative manipulation tasks without giving specific metrics.Source: §1 · p. 3

2 DUOMIND: DISTRIBUTED HIERARCHICAL MULTI-AGENT FRAMEWORK

DuoMind distributes coordination across robots by separating high-level VLM reasoning from low-level VLA execution and using natural language as a shared interface. Structured peer messages convey information about actions, expected states, task beliefs, and uncertainty.

2 DUOMIND: DISTRIBUTED HIERARCHICAL MULTI-AGENT FRAMEWORK

  • At each planning step, every robot’s VLM orchestrator reasons over the task instruction, local observation, and peer messages, then issues a low-level instruction and a structured message for other robots.The VLA follows the instruction to produce fine-grained actions.Source: Figure 2 · p. 3, §2 · p. 3

2.1 MULTI-AGENT COORDINATION VIA EMBODIED REASONING

  • The orchestrator determines a robot’s current subtask and translates it into a short-horizon instruction, leaving the VLA to generate fine-grained actions from local observations.This separation assigns multi-agent understanding and coordination to the orchestrator and low-level control to the action model.Source: §2.1 · p. 4
  • This decomposition avoids requiring action models, primarily trained on single-agent demonstrations, to interpret complex multi-agent state; existing foundation models can be reused largely off the shelf with lightweight adaptation.The paper focuses on two-agent settings because task design, data collection, evaluation, and computation costs increase with team size; larger teams are left for future study.Source: §2.1 · p. 4

2.2 BRIDGING HIGH-LEVEL REASONING AND LOW-LEVEL CONTROL VIA NATURAL LANGUAGE BASED SEMANTIC COMMUNICATION

  • Natural language serves as a common interface between orchestrator and action model, permitting off-the-shelf components with minimal adaptation and independent replacement.Compared with interfaces that require tighter coupling, the paper emphasizes language’s interpretability and compatibility with language-conditioned models.Source: §2.2 · p. 4
  • The shared linguistic interface also lets heterogeneous agents interpret one another’s messages and allows users to inspect or revise instructions in ordinary language.The paper describes this as a foundation for coordination among heterogeneous robots and multiple human users.Source: §2.2 · p. 4

2.3 INTER-AGENT INFORMATION SHARING

  • To address partial observability and changing task states, agents exchange structured natural-language messages that add information about decisions and task-relevant environmental changes.The messages provide additional context for high-level reasoning.Source: §2.3 · pp. 4–5
  • Messages specify the sender’s intended next subtask and expected resulting state, helping peers avoid interference and act proactively.The intention matches the low-level instruction sent to the action model; the subgoal describes the state expected after that subtask.Source: §2.3 · p. 5
  • Messages can also share observed or inferred task beliefs and an optional uncertainty signal, giving orchestrators context for weighing ambiguous or inconsistent information.The paper states that this compact context supports more precise low-level instructions and safer, more efficient coordination.Source: §2.3 · p. 5

3 ROBOPOLY

RoboPoly is a benchmark for long-horizon manipulation tasks designed to require coordination between independent robots under distributed execution. It includes seven tasks, shared two-robot workspaces, locally constrained observations and control, and generated trajectory and language-instruction data.

3 ROBOPOLY

  • RoboPoly targets long-horizon manipulation requiring explicit cooperation between two independent robots, unlike RoboTwin’s dual-arm setting, which does not necessarily require cooperation.The benchmark emphasizes distributed execution, with each robot acting on its own observations and policy.Source: §3 · p. 5

3.1 TASKS FOR MULTI-ROBOT COORDINATION

  • The seven tasks are Food Serve, Prepare Snack, Clean Table, Cook Pot, Put Object Cabinet, Hang Bag, and Exchange Bread.Workspace and object placement are designed so a single robot cannot always complete a task independently or efficiently.Source: §3.1 · p. 5

3.2 OBSERVATIONS, ACTIONS, AND ROBOTS

  • Each robot receives the global camera view and its own wrist-camera view, but not the other robot’s wrist view, and controls only its own 7-DoF Franka Panda arm.Key objects are randomized within predefined task-specific regions at episode start.Source: §3.2 · pp. 5–6

3.3 DATASETS

  • The dataset pipeline plans trajectories between key manipulation states, replays them to render camera observations, and pairs them with robot states, actions, and high- and low-level instructions.The authors state that the resulting dataset is open-sourced for multi-robot coordination research.Source: §3.3 · p. 6

4 EXPERIMENTS

DuoMind is evaluated under distributed control on RoboPoly and RoboTwin, where it matches or outperforms baselines across most tasks. The experiments examine hierarchical orchestration, inter-agent communication, and compatibility with different action models.

4 EXPERIMENTS

  • Evaluation covers seven RoboPoly and eight RoboTwin tasks under distributed control, with 400 rollouts per task and success rates reported for DuoMind and baselines.RoboTwin's dual-arm setting is adapted by treating the arms as independent robots with separate control.Source: §4 · p. 6, §4.1 · p. 7

4.2 MAIN RESULTS

  • DuoMind matches or outperforms baselines on most tasks across seven RoboPoly and eight RoboTwin tasks.The tables report success rates; RoboPoly tasks generally have longer horizons and lower overall success rates, while DuoMind's improvement over baselines is more significant there.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · pp. 7–8
  • The orchestrator adapts subtasks and low-level instructions to evolving states and peer messages, helping robots avoid conflicts and recover after unexpected interactions.Fixed high-level instructions can leave action policies uncertain after execution errors or when one robot changes another's workspace.Source: §4.2 · pp. 7–8

4.3 ABLATION OF INTER-AGENT COMMUNICATION

  • Disabling peer messages increases conflicts and asynchronous behavior, degrading coordination under distributed observations.The ablation disables messages while retaining the evaluation setting used for the main results.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 ACTION MODEL COMPATIBILITY

  • DuoMind with π0 has lower success rates than with π0.5, but substantially outperforms the π0-only baseline.The framework separates high-level orchestration from low-level control, allowing different action models to be integrated.Source: §4.4 · p. 9

5 RELATED WORK

Prior work addresses decentralized coordination, generalist robot policies, and multi-agent benchmarks, but DuoMind combines distributed reasoning and natural-language communication with VLA-based execution. RoboPoly focuses on long-horizon coordination and closed-loop execution under distributed observation and control.

5.1 LEARNING-BASED MULTI-AGENT COORDINATION

  • Classical multi-agent learning often requires task-specific joint training and uses latent coordination that is difficult to interpret or integrate with pretrained robot foundation models.DuoMind instead uses decentralized reasoning with explicit natural-language exchange of task-relevant information.Source: §5.1 · p. 9

5.2 ROBOT LEARNING AND MULTI-ROBOT PLANNING

  • Hierarchical robot policies and multi-robot systems commonly focus on single-robot execution or rely on predefined skills, symbolic actions, waypoints, or motion planners.DuoMind combines distributed high-level reasoning and communication with VLA-based physical execution.Source: §5.2 · p. 9

5.3 ROBOT LEARNING AND MULTI-AGENT BENCHMARKS

  • Existing robot-learning benchmarks largely emphasize single robots or centralized embodiments, while multi-agent benchmarks often separate abstract planning from low-level manipulation.RoboPoly evaluates long-horizon coordination and closed-loop execution under distributed observation and control.Source: §5.3 · p. 9

6 CONCLUSION

DuoMind combines high-level reasoning, low-level control, and semantic inter-agent communication for multi-robot coordination, and RoboPoly evaluates long-horizon manipulation under distributed observations and control. Experiments on RoboPoly and RoboTwin support the value of hierarchical orchestration and explicit information sharing, while the decoupled design permits model integration.

6 CONCLUSION

  • DuoMind combines high-level reasoning, low-level control, and semantic communication, while RoboPoly evaluates long-horizon multi-robot manipulation under distributed observations and control.Experiments on RoboPoly and RoboTwin show that hierarchical orchestration improves execution and that explicit information sharing matters under partial observability.Source: §6 · p. 9
  • The decoupled framework can incorporate stronger VLMs and VLAs, with real-world multi-robot coordination identified as future work.The paper describes more capable and efficient real-world coordination as a promising direction.Source: §6 · p. 10

A TASK SETTINGS OF ROBOPOLY

RoboPoly contains seven tasks requiring two robots to coordinate around reach limits, object transfers, simultaneous work, or joint manipulation. Tasks span food preparation, object handling, and transport.

TASKS

  • Food Serve and Exchange Bread require passing bread between robots because one robot cannot reach both the source and target locations.Food Serve places two bread pieces and a tray across the workspace; Exchange Bread places target boxes beyond each robot's reachable workspace.Source: A TASK SETTINGS OF ROBOPOLY · pp. 13–14
  • Prepare Snack requires placing four bread pieces on two plates while coordinating simultaneous motions and avoiding collisions; Clean Table requires transferring a cube when the container is out of reach.Both tasks distribute objects or destinations across the shared workspace.Source: A TASK SETTINGS OF ROBOPOLY · p. 13
  • Cook Pot requires sequential lid and carrot handling followed by a joint transport, while Put Object Cabinet requires opening, loading, and closing a drawer.Hang Bag requires one robot to pass a bag to the other robot for hanging on a hook.Source: A TASK SETTINGS OF ROBOPOLY · pp. 13–14

B EFFECT OF INTER-AGENT COMMUNICATION

Disabling peer messages leaves each orchestrator planning from local observations and task instructions; the reported failures are coordination conflicts that communication is intended to reduce.

B EFFECT OF INTER-AGENT COMMUNICATION

  • The ablation disables peer messages so each orchestrator generates instructions using only its local observation and the task instruction.The study uses DuoMind with π0.5 as the action model and otherwise follows the section 4.2 evaluation setting.Source: B EFFECT OF INTER-AGENT COMMUNICATION · p. 14
  • Without inter-agent communication, the robots more often conflict or act asynchronously, and explicit messaging provides a mechanism for sharing information to reduce those conflicts.The authors also report improved synchronized and safe coordination with communication.Source: B EFFECT OF INTER-AGENT COMMUNICATION · pp. 14–15
  • In Cook Pot, one robot may start lifting before the other has grasped its handle, causing coordinated manipulation to fail.In Prepare Snack, both robots may target the same plate and collide while placing bread.Source: B EFFECT OF INTER-AGENT COMMUNICATION · p. 15

C ACTION MODEL COMPATIBILITY

DuoMind accommodates different action models: the π0 version performs below the π0.5 version but remains substantially better than the π0-only baseline, showing the orchestrator contributes beyond the action model.

C ACTION MODEL COMPATIBILITY

  • DuoMind can integrate different action models without changing its coordination framework, while stronger action models can further increase overall success rates.The orchestrator supports high-level reasoning and coordination, while a stronger action model provides more reliable and stable low-level execution.Source: C ACTION MODEL COMPATIBILITY · pp. 15–16
  • The comparison indicates that the performance gain is not solely determined by the action model; the orchestrator provides high-level reasoning and subtask decomposition.It generates instructions suited to the evolving multi-agent state, improving coordination over the action-model-only setting.Source: C ACTION MODEL COMPATIBILITY · p. 15
  • DuoMind with π0 has lower success rates than DuoMind with π0.5, yet substantially outperforms the π0-only baseline.The comparison uses different action models within DuoMind.Source: Figure 8 · p. 15, C ACTION MODEL COMPATIBILITY · p. 15

D TRAINING AND EVALUATION CONFIGURATIONS

The configuration description reports fine-tuning and evaluation settings for RoboPoly and RoboTwin, including rollout counts, camera inputs, action-chunk timing, and the learning-rate schedule.

D TRAINING AND EVALUATION CONFIGURATIONS

E EVALUATION VISUALIZATION

The evaluation visualizations cover selected RoboPoly and RoboTwin tasks, while the orchestrator prompt specifies how each robot plans, communicates, and advances subgoals.

E EVALUATION VISUALIZATION

F VLM SYSTEM PROMPT

  • The orchestrator prompt assigns high-level planning and message generation, using workspace reachability, peer intention, task state, and the robot’s intention, belief, and uncertainty.It selects complementary subgoals and sends concise structured information to the peer.Source: F VLM SYSTEM PROMPT · p. 18
  • For staged candidate instructions, the orchestrator starts at the lowest-numbered stage, repeats it until visibly complete, then advances by at most one stage.It must not repeat completed subgoals, advance merely with time, or accept the other agent’s completion claim without visual verification.Source: F VLM SYSTEM PROMPT · p. 19
  • The prompt takes the task, global-camera image, prior instruction and planning history, and peer message as inputs, then returns a low-level instruction and shared message.The shared message contains intention, subgoals, and belief.Source: F VLM SYSTEM PROMPT · p. 19
Loading 2610.02161v1…