Co^2Gen: Collaboratively Generating Task Plans for Decentralized Multi-Robot and Human Collaboration

I. INTRODUCTION

• VLA models finetuned on single-robot demonstrations fail to coordinate on collaborative manipulation tasks.The authors relate this to pretrained data dominated by non-collaborative single-robot demonstrations.Source: §I · p. 1
• Collecting simultaneous multi-robot demonstrations can require additional human labor or unavailable teleoperation interfaces.Source: §I · p. 1
• Co2Gen uses decentralized collaborative MCTS to generate executable Pythonic plans, with VLMs verifying skill preconditions.Plans compose and call individual robots’ available skills.Source: §I · p. 1
• The generated plans produce collaborative demonstrations with a human task-success verifier, then each robot’s VLA is finetuned separately from its local data stream.Source: §I · pp. 1–2
• Co2Gen-generated demonstrations let decentralized VLA finetuning match human-teleoperated task performance and retain that match in zero-shot human-robot collaboration.The paper also reports improved collaborative-task performance after finetuning on collaborative demonstrations.Source: §I · pp. 1–2

II. RELATED WORK

• Prior robot-planning approaches include language-to-skill selection [1], code generation for single-robot policies, and centralized or dialog-based multi-robot planning.Source: §II · p. 2
• Earlier multi-robot VLA approaches use information sharing between arms, while this work targets separate VLA policies for decentralized collaboration and zero-shot human collaboration.The authors describe this as extending bi-manual manipulation to a true multi-robot system.Source: §II · p. 2

III. PROBLEM FORMULATION

• The problem formulation seeks a collaborative plan from a natural-language task prompt that generates successful, varied episodes for VLA finetuning.Source: §III · p. 2
• Each robot has a set of single-robot skills, and those sets may contain shared or robot-specific skills.Source: §III · p. 2
• The experiments focus on a dual-arm collaboration setup.Source: §III · p. 2

A. Robot Setup

• Experiments use a bi-manual low-cost SO-101 arm with an overhead stereo Zed2 camera and a wrist camera near each gripper.Source: A. Robot Setup · p. 2

IV. METHOD

• The bi-manual arm setup is used for all collaborative tasks.Source: Figure 2 · p. 3

A. Collaborative Plan Generation Module.

• Plans identify sub-task preconditions, allocate skills to robots, and schedule execution so one robot can wait for another’s successful sub-task.Code-based plans condition low-level skills on environmental feedback and support decentralized parallel execution.Source: A. Collaborative Plan Generation Module · pp. 2–3
• Two VLM agents, an actor and a critic, collaboratively refine an executable Pythonic plan through Monte Carlo Tree Search.Both receive the overhead-camera observation, task prompt, and available single-robot skills; the critic scores the actor’s proposals and gives targeted feedback.Source: A. Collaborative Plan Generation Module · p. 3
• UCT guides tree traversal to a leaf, where the module expands the plan into n children refined from their parent’s critique.The selection score uses the node’s back-propagated subtree value, visit count, and an exploration coefficient.Source: A. Collaborative Plan Generation Module · p. 3
• The critic scores each plan on progress, task distribution, and precondition checking, with each criterion scored 1–5 and a maximum total of 15.Source: A. Collaborative Plan Generation Module · p. 3
• Search stops at maximum depth dmax; the system selects the highest-value solved node above rthres, or falls back to the node with the highest back-propagated value.During backpropagation, the critic score updates each node’s visitation count and value along the traversal path.Source: A. Collaborative Plan Generation Module · p. 3

B. Task Execution Module

• VLMs query task preconditions from the overhead and wrist cameras during execution of the collaborative plan.Source: B. Task Execution Module · p. 3
• A perception module estimates object coordinates using hybrid blob detection and YOLOv8 via the overhead depth camera, and an inverse-kinematics solver plans trajectories.Source: B. Task Execution Module · p. 4
• The robot arms use wrist cameras to obtain precise object coordinates for grasping.Source: B. Task Execution Module · p. 4

C. Collaborative Data Generation

• A human verifier checks whether each task execution succeeded before its episode is added to Dtask.Source: C. Collaborative Data Generation · p. 4
• Human verification is chosen to mitigate false-positive trajectories from VLM verification errors or hallucinations.Source: C. Collaborative Data Generation · p. 4

D. Decentralized VLA Finetuning

• Each robot’s VLA is finetuned decentrally on its own data stream, which contains that robot’s joint state and wrist-camera stream and shares the overhead stereo-camera stream.Source: D. Decentralized VLA Finetuning · p. 4
• The hybrid training approach combines collaborative task data with single-robot human demonstrations to pair coordination signals with trajectory diversity.The authors note that single-robot demonstrations lack collaborative coordination, while collaborative trajectories may lack human-demonstration diversity.Source: D. Decentralized VLA Finetuning · p. 4

V. EXPERIMENTS

• The experiments ask how finetuning on human and Co2Gen collaborative demonstrations affects decentralized task performance and performance with novel human partners in zero-shot.Source: §V · p. 4
• The generated plan uses natural-language prompts to check preconditions before subsequent sub-tasks, supporting coordination during execution.Source: Figure 4 · p. 5

• The storage task places a pull-out cabinet between the arms and asks them to store a colored block; stacking asks them to combine blocks centrally.Each task requires both arms to coordinate their assigned subtasks.Source: A. Evaluation Tasks · p. 5
• In storage, role 1 opens and closes the cabinet, while role 2 places the cube inside.Source: A. Evaluation Tasks · p. 5
• In stacking, role 1 places the bottom cube and role 2 stacks the top cube.Source: A. Evaluation Tasks · p. 5

B. Implementation Details

• Each task uses 50 demonstration episodes and 20,000 finetuning steps; DAgger adds correction data, with episodes split by role.The setup uses GPT 5.6 for plan generation and precondition checking, and SmolVLA as the finetuned model.Source: B. Implementation Details · p. 5

C. Decentralized Collaborative Task Execution with VLAs

• The evaluation compares Co2Gen-trained VLAs, with and without added single-arm data, against single-arm and bimanual human-demonstration baselines.Policies are evaluated over 20 trials and run until task completion, irrecoverable failure, or timeout.Source: C. Decentralized Collaborative Task… · pp. 5–6
• Evaluation reports overall task success rate and task score, which reflects successfully completed subtasks scored according to Table I.A subtask counts as successful when its assigned arm executes it without collision with the other arm.Source: C. Decentralized Collaborative Task… · p. 6, Table I · p. 6
• Single-arm demonstrations produce the lowest task performance, with arms failing to coordinate cabinet access or stacking order.Source: C. Decentralized Collaborative Task… · p. 6
• Bimanual-data VLAs significantly outperform single-arm-data VLAs on both tasks, but their task success rates remain below 50%.The authors hypothesize that the models do not capture full coordination signals without information exchange during training.Source: C. Decentralized Collaborative Task… · p. 6
• Co2Gen data improves over single-arm data, but its storage-task subtask success rate drops off.Source: C. Decentralized Collaborative Task… · p. 6
• Co2Gen-trained VLAs match human-teleoperated VLA task performance on storage within error and exceed it on stacking.The figure reports task scores, subtask success rates, and overall task success rates across 20 executions.Source: Figure 7 · p. 7, C. Decentralized Collaborative Task… · p. 7

• Humans may detect and respond to coordination signals better than autonomous partners, allowing collaboration even with a single-arm-trained VLA in stacking role 1.Source: D. Human-Robot Collaboration · p. 7
• Collaborative data remains important: single-arm-trained VLAs perform poorly in stacking role 2 and both storage roles.Source: D. Human-Robot Collaboration · p. 7

VI. CONCLUSION

• VLAs finetuned on Co2Gen data match the task performance of VLAs finetuned on human-teleoperated collaborative data.The conclusion presents generated collaborative plans as a substitute for human teleoperation in producing multi-robot training data.Source: §VI · p. 7
• Co2Gen uses collaborative MCTS to generate Python functions that plan multi-robot execution and VLMs to verify sub-task preconditions.The framework uses these plans to generate collaborative data for decentralized VLA finetuning without human teleoperated collaborative data.Source: §VI · p. 7
• Future work targets humanoids and mobile robots, longer-horizon collaborative tasks, complex environments, and larger teams.Source: §VI · p. 7