Source-linked AI summary
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Zhao Mandi, Shreeya Jain, Shuran Song
TL;DR
RoCo addresses the challenge of coordinating multi-robot systems across heterogeneous capabilities and difficult motion-planning spaces. It uses LLM-based dialogue to produce validated sub-task plans and waypoint paths, then applies centralized multi-arm planning. Across RoCoBench, the paper reports generality across collaboration scenarios and adaptation to varying task semantics, while identifying perception and LLM-query constraints.
Problem
Multi-robot collaboration requires task-level coordination across robot capabilities, collision-free motion planning, and adaptation beyond task-specific engineering.
Method
RoCo equips robots with LLM agents for dialogue, generates feedback-improved sub-task plans and waypoints, and passes validated goals to a centralized multi-arm motion planner.
Results
RoCo demonstrates generality across six RoCoBench tasks, including flexibility across collaboration scenarios and adaptation to varying task semantics without task-specific training.
Takeaways & Limitations
RoCo provides an interpretable and flexible framework for multi-robot coordination, with experiments showing dialogue can support collaboration with a human user.
Takeaways & Limitations
RoCo assumes accurate perception, and repeated LLM queries can be costly, delay responses, and limit suitability for dynamic or speed-sensitive tasks.
Abstract
from arXiv · showhide
We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning. Robots are equipped with LLMs to discuss and collectively reason task strategies. They then generate sub-task plans and task space waypoint paths, which are used by a multi-arm motion planner to accelerate trajectory planning. We also provide feedback from the environment, such as collision checking, and prompt the LLM agents to improve their plan and waypoints in-context. For evaluation, we introduce RoCoBench, a 6-task benchmark covering a wide range of multi-robot collaboration scenarios, accompanied by a text-only dataset for agent representation and reasoning. We experimentally demonstrate the effectiveness of our approach -- it achieves high success rates across all tasks in RoCoBench and adapts to variations in task semantics. Our dialog setup offers high interpretability and flexibility -- in real world experiments, we show RoCo easily incorporates human-in-the-loop, where a user can communicate and collaborate with a robot agent to complete tasks together. See project website https://project-roco.github.io for videos and code.
1 Introduction
RoCo uses LLM agents to coordinate multi-robot tasks through dialogue, validated sub-task plans, and centralized multi-arm motion planning. The framework targets task understanding, collision-free planning, and adaptation without task-specific engineering.
- Motivation: Multi-robot systems must coordinate heterogeneous capabilities while avoiding increasingly difficult collision-free planning in growing configuration spaces.Traditional systems also rely on task-specific engineering, limiting adaptation to new scenarios and task variations.
- Multi-agent dialog: RoCo delegates each robot to an LLM agent that discusses task strategy and exchanges role-specific information in natural language.The shared prompt structure includes task context, dialogue history, capabilities, communication instructions, observations, and optional plan feedback.
- Contributions: RoCo combines LLM communication, sub-task planning, and motion planning to handle varied collaboration scenarios without task-specific training.The framework also introduces RoCoBench to evaluate task-level coordination and action-level motion planning across different semantics, overlaps, capabilities, and embodiments.
- Validated planning: Dialog produces per-agent sub-task plans and optional 3D waypoint paths, which are checked for formatting, task constraints, IK feasibility, collisions, and waypoint validity.Failed validations are fed back to the agents for replanning, subject to a maximum number of attempts.
- Motion planning: Validated plans yield joint-space goal configurations that a centralized RRT-based multi-arm planner converts into trajectories for all robots.The resulting trajectories are executed by the robots’ arm joint controllers.
4 Benchmark
RoCoBench is a six-task tabletop manipulation benchmark designed to test multi-robot coordination across diverse task structures and information settings.
- Benchmark scope: RoCoBench contains 6 multi-robot collaboration tasks in a tabletop manipulation setting with semantically understandable common-sense objects.The tasks cover a repertoire of collaboration scenarios requiring different communication and coordination behaviors.
- Task decomposition: Three tasks are sequential and three are parallel, testing whether subtasks must follow an order or can be completed concurrently.Make Sandwich requires ordered stacking, whereas Pack Grocery permits objects to be placed in any order.
- Observation space: The benchmark varies how much task and environment information each robot receives through differences in observation space.This property evaluates coordination when agents have unequal information.
5 Experiments
RoCoBench evaluates dialogue-based coordination, feedback-informed planning, waypoint generation, semantic adaptation, and human-robot collaboration across simulated and real-world experiments. Results indicate that full prompting performs best overall, LLM-proposed waypoints mainly benefit placing, and RoCo adapts to varied task semantics.
- 5.1 Main Results on RoCoBench: RoCoBench spans tasks differing in decomposition, observation sharing, and workspace overlap, enabling systematic evaluation of collaboration behaviors.Task decomposition distinguishes parallel from sequential subtasks; observation space distinguishes shared from asymmetric information; workspace overlap captures robot proximity during execution.
- 5.1 Main Results on RoCoBench: The main comparison evaluates Dialog against Central Plan and ablations removing dialog history or environment feedback, using success, efficiency, and replanning metrics.Central Plan receives full observations and capabilities, while the ablations alter prompt history or feedback and adjust episode budgets.
- 5.1 Main Results on RoCoBench: Full prompting achieves the best overall results, while dialog agents sometimes match the oracle despite receiving less information.In Sort Cubes, dialog helps agents find a cooperative strategy, whereas the oracle makes mistakes satisfying all agents’ constraints simultaneously.
- 5.2 Effect of LLM-proposed 3D Waypoints: LLM-proposed waypoints significantly accelerate placing planning but show no clear benefit for picking compared with linear and hard-coded alternatives.The comparison uses Pack Grocery and Move Rope, whose high workspace overlap makes collisions between arms and desktop objects more likely during placing.
- 5.3 Zero-shot Adaptation to Task Variations: RoCo adapts to randomized object locations, changed sandwich recipes, and reach-limited robot capabilities without task-specific re-programming.Agents reason about initialization, coordinate subtask strategies for different goals, and exchange reach-related information when planning.
- 5.4 Real-world Experiments: Human-Robot Collaboration: The real-world study uses a robot arm and human collaborator for block sorting, varying initialization, task order, and whether the human corrects perception and response errors.Ten runs are evaluated for each setup, with success and steps reported; incorrect OWL-ViT detections primarily bottleneck performance.
6 Multi-Agent Representation and Reasoning Dataset
RoCoBench evaluation data is curated into a text-only dataset testing multi-agent representation and reasoning without robotic-environment interaction. GPT-4 leads across categories, although performance remains below full accuracy.
- Dataset: RoCoBench-Text evaluates agent representation and task reasoning without requiring interaction with a robotic environment.It extends RoCoBench evaluation data with open-ended questions beyond selecting the next action plan.
- Dataset: The dataset tests self-knowledge, memory retrieval, and communication skills using yes/no, multiple-choice, and short-answer questions.Self-knowledge includes capability understanding and retrieval of information from prior dialogue and actions.
- Evaluation: The benchmark includes questions requiring agents to recognize their capabilities and answer reachability queries.One example asks which cubes the robot can reach given its current access panels.
- Results: GPT-4 leads performance across all evaluated categories, with only small variations between its two tested versions.The evaluation compares GPT-4, GPT-3.5-turbo, Claude-v1, and two GPT-4 timestamps.
- Results: Despite leading across categories, GPT-4 remains considerably short of fully accurate performance.The authors position the dataset as useful for future improvement and evaluation of language models.
7 Limitation
RoCo’s limitations concern perception assumptions, open-loop execution, and reliance on repeated LLM queries. These constraints affect robustness to imperfect sensing, execution errors, and time-sensitive tasks.
- Perception: RoCo assumes accurate perception, so object-detection errors in real-world experiments can cause planning mistakes.The simulation uses oracle state information, while imperfect pretrained detection exposes this limitation in physical experiments.
- Execution: Open-loop trajectory execution can produce errors that LLMs cannot recognize or handle through abstract scene and action descriptions.The limitation applies after the planner has generated trajectories for robot execution.
- Efficiency: Querying pretrained LLMs for every dialogue response can be expensive and introduce delays unsuitable for dynamic or speed-sensitive tasks.The authors identify both monetary cost and model reaction time as concerns.
- Scope: The approach depends on combining LLM planning with robotic perception and motion-planning components rather than resolving perception limitations itself.Related work identifies perception as a bottleneck for combining LLMs with robotics.
- Scope: Unlike prior single-question dialogue or debate settings, RoCo must plan separate actions for each agent and reach consensus.This adds complexity to the discussion process.
- Scope: RoCo’s evaluation focuses mainly on static scenes, while multi-robot motion planning also includes more challenging dynamic settings.The cited related-work discussion distinguishes RoCo’s task setting from broader motion-planning research.
9 Conclusion
RoCo is presented as a general LLM-based framework for multi-robot coordination and planning, alongside RoCoBench for systematic evaluation. The experiments show adaptation to varying task semantics while exposing limitations for future work.
- Conclusion: RoCo is a framework that uses LLMs for multi-robot coordination and planning.The authors describe it as a new framework for collaboration across robots.
- Conclusion: RoCoBench provides a 6-task benchmark for multi-robot manipulation intended for broader research evaluation.The benchmark is introduced as an open resource for the research community.
- Conclusion: Experiments demonstrate generality and few-shot adaptation to varying task semantics, while identifying limitations and room for improvement.The conclusion frames these findings as opportunities for future research.
10 RoCoBench
RoCoBench is a MuJoCo-based suite of six simulated multi-robot collaboration tasks spanning sweeping, sandwich making, cube sorting, and grocery packing. The tasks vary robot capabilities, reachability, action structure, and collision-avoidance demands.
- Benchmark: RoCoBench is built with the MuJoCo physics engine and documents six simulated collaboration tasks.The benchmark uses open-source robotics assets and related efforts in its task development.
- Sweep Floor: Sweep Floor requires two robots to position a dustpan and broom around cubes, sweep them into the dustpan, and dump them into a trash bin.Alice holds the dustpan and Bob holds the broom, with MOVE, SWEEP, WAIT, and DUMP skills.
- Make Sandwich: Make Sandwich requires two robots with different ingredient access to select items and take turns stacking them in the correct order.Chad reaches the right side and Dave reaches the left side; available skills are PICK, PUT, and WAIT.
- Benchmark design: The benchmark therefore combines sequential tasks, parallelizable tasks, asymmetric capabilities, and close-proximity coordination demands.Make Sandwich is sequential, while Pack Grocery objects may be placed in any order.
- Sort Cubes: Sort Cubes assigns three robots distinct target panels and reach ranges, requiring assistance to move cubes closer before placement.Alice, Bob, and Chad each have different target panels and overlapping neighboring reach ranges.
- Pack Grocery: Pack Grocery asks two robots to move nearby grocery items into a bin while coordinating paths to avoid collisions.The task gives Alice broad pick-and-place access to table objects.
2. Franka Panda (‘Bob’): can pick and place any object on the table
The benchmark includes tasks requiring robots to observe workspace state, use constrained skills, and coordinate manipulation across shared workspaces. Examples include jointly moving a rope and arranging cabinet contents.
- Move Rope: The Move Rope task requires two robots to lift a rope over a wall, place it in a groove, and coordinate grippers to avoid collision.
- Move Rope: Alice and Bob can each pick and place a rope end within their respective reachable workspace.
- Move Rope: The rope task exposes gripper, rope-end, wall-corner, and groove-end locations to the agents.
- Robot skills: Available skills include task-space waypoint paths for picking and placing in rope-related tasks, while cabinet tasks add door opening and waiting.
- Arrange Cabinet: The Arrange Cabinet task uses three robots: two hold the cabinet doors while the third moves cups and mugs onto correct coasters.
- Arrange Cabinet: Arrange Cabinet observations include door-handle locations and each robot’s reachable objects, without revealing other robots’ reach ranges.
11 Details on LLM Prompting
RoCo uses structured multi-agent prompting to turn dialog into validated sub-task plans and executable robot trajectories. Dialog history, observations, capabilities, and feedback are repeatedly incorporated into this process.
- Implementation: The implementation uses a separate LLM query for every agent response in the dialog.
- Prompt structure: Each robot receives a prompt describing its identity, task context, capabilities, history, current observation, and coordination instructions.
- Multi-agent dialog: Algorithm 1 maintains dialog and feedback buffers while iterating through bounded dialog rounds and re-planning attempts.
- Plan refinement: After dialog, agents generate a plan that is validated; failed plans produce feedback for another dialog round, while valid plans are parsed as the final plan.
- Execution: The final plan is passed to a motion planner, executed in the environment, and followed by the next observation and reward.
12 Additional Experiment Results
The additional experiments examine failure modes and a human collaboration setup. They show that dialog can propagate an incorrect completion belief, while open-loop execution can turn small trajectory errors into physical mistakes.
- Failure analysis: A mistaken agent completion claim can cause subsequent agents to repeat the error and wait even though the task remains unfinished.
- Failure analysis: Open-loop trajectory execution means small motion errors can produce unexpected outcomes such as accidentally knocking over an object.
- Real-world setup: The real-world setup uses a 6DoF UR5E with suction gripper and GPT-4 to control robot agent Bob in dialog with human Alice.
- Real-world setup: Bob’s action options are to pick a block and place it in the wooden bin, or wait for Alice to move blocks from cups to the table.
- Human collaboration: The robot prompt specifies that Bob cannot pick blocks inside cups and must wait for Alice before picking table-top blocks.
13 Toy Example on LLM for 3D Path Planning
The toy experiment evaluates GPT-4’s ability to plan collision-free multi-agent paths in randomly generated 3D grids and revise failed plans using feedback. Performance varies with temperature and improves through re-planning.
- Experimental setup: Each evaluation samples a new obstacle layout and agent start-goal configuration, with up to five attempts using appended feedback after failures.
- Prompting: GPT-4 is prompted with obstacle coordinates, agent initial and goal coordinates, and any previous failed plan to produce a collision-free path.
- Results: A moderate temperature around 0.6 gives the best success rate and the most one-shot successes across 30 evaluation runs.
- Results: GPT-4 can use collision feedback to correct a path that failed on an earlier attempt.
- Results: Figure 13 compares planning success across temperatures on size 53 and size 103 3D grids, with lower success on the more challenging larger grid.
14 Multi-Agent Representation and Reasoning Dataset
The dataset evaluates multi-agent representation and reasoning across diverse collaboration tasks, including sequential transport, sandwich making, floor sweeping, and cabinet arrangement. Questions test coordination, spatial reasoning, task state understanding, and robot-to-robot assistance.
- Dataset scope: The dataset spans sequential transport, sandwich making, floor sweeping, and cabinet arrangement tasks.The passages report task names and question counts for these scenarios.
- Dataset composition: The dataset includes 57 questions for sequential transport, 44 for sandwich making and floor sweeping, and 41 for cabinet arrangement.These counts are reported alongside the corresponding task descriptions.
- Reasoning context: Task contexts encode collaboration constraints such as robot reachability, workspace sides, object locations, and required action ordering.Examples specify which panels or food items each robot can reach and impose ordering or turn-taking requirements.
- Agent reasoning: Prompts test whether agents can infer task state and coordinate assistance across dialogue history and current goals.Examples ask agents to recover another robot’s initial objects or decide how to help complete a teammate’s placement task.
- Question formats: Questions use multiple formats, including binary yes-or-no, multiple-choice, and open-ended responses.The reported formats include 96 binary questions, 31 multiple-choice questions with options A–C, and contextual reasoning prompts.