PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperation

Haojun Shi, Suyu Ye, Katherine M. Guerrerio, Jianzhi Shen, Yifan Yin, Daniel Khashabi, Chien-Ming Huang, Tianmin Shu

arXiv:2602.19304v1cs.ROcs.AIcs.HCcs.MA

TL;DR

Decentralized agents must adapt paths to one another, but may not know others’ intentions, making language communication relevant to safe cooperation. CaPE uses a VLM to synthesize edits to planner-proposed paths and verifies them with a planner. Across simulated and real-world scenarios, it is reported to generalize and outperform baselines while maintaining safety and interpretability.

  • Problem

    Decentralized agents may not know one another’s goals or plans, and grounding language communication in robot path planning remains challenging.

    Source: §I · p. 1

  • Method

    CaPE uses a VLM to synthesize path-editing programs from multimodal context and verifies edits with a model-based planner before execution.

    Source: §I · p. 2, 3) Path Editing Program Synthesis via… · p. 3

  • Results

    Across simulated and real-world scenarios, CaPE generalizes to unseen layouts, novel tasks, different agent counts, and open-ended instructions, outperforming baselines while maintaining safety and interpretability.

    Source: §I · p. 2, §VI · p. 9

  • Takeaways & Limitations

    CaPE presents language as guidance for verifiable edits to existing plans, rather than direct generation of paths or low-level actions.

    Source: §VI · p. 9

    Safety checks can reject infeasible edits, so the resulting plan may not exactly reflect the human’s original intent.

    Source: C. Limitations and future work · pp. 8–9

Abstract

from arXiv · show

Successful cooperation among decentralized agents requires each agent to quickly adapt its plan to the behavior of other agents. In scenarios where agents cannot confidently predict one another's intentions and plans, language communication can be crucial for ensuring safety. In this work, we focus on path-level cooperation in which agents must adapt their paths to one another in order to avoid collisions or perform physical collaboration such as joint carrying. In particular, we propose a safe and interpretable multimodal path planning method, CaPE (Code as Path Editor), which generates and updates path plans for an agent based on the environment and language communication from other agents. CaPE leverages a vision-language model (VLM) to synthesize a path editing program verified by a model-based planner, grounding communication to path plan updates in a safe and interpretable way. We evaluate our approach in diverse simulated and real-world scenarios, including multi-robot and human-robot cooperation in autonomous driving, household, and joint carrying tasks. Experimental results demonstrate that CaPE can be integrated into different robotic systems as a plug-and-play module, greatly enhancing a robot's ability to align its plan to language communication from other robots or humans. We also show that the combination of the VLM-based path editing program synthesis and model-based planning safety enables robots to achieve open-ended cooperation while maintaining safety and interpretability.

I. INTRODUCTION

Decentralized agents need to adapt paths despite uncertainty about others’ goals and plans, and grounding language in robot path planning remains challenging. CaPE frames this as language-guided path editing, using planner-verified VLM programs for cooperation across robot and human-robot settings.

I. INTRODUCTION

  • Agents often lack knowledge of one another’s goals or plans, while optimal multi-agent path planning is NP-hard even for centralized planners in discrete spaces.The paper identifies grounding language communication in robot path planning as an additional challenge.Source: §I · p. 1
  • CaPE treats communication as guidance for editing existing paths, with planner verification intended to preserve safety while supporting efficient adaptation.The framework is presented for human-robot and multi-robot cooperation, including joint carrying and autonomous driving.Source: §I · pp. 1–2

1) Program Synthesis for Robots:

2) Communicative Multi-Agent Cooperation:

  • Prior communicative multi-agent work emphasizes high-level coordination, whereas CaPE uses language to edit paths under physical and safety constraints.The cited prior-work categories include task allocation, role negotiation, and subgoal planning [60] [51] [49] [37].Source: 2) Communicative Multi-Agent Cooperation · p. 2

3) Multi-Agent Path Planning:

  • Centralized planners restrict settings where humans or independently controlled robots are outside the optimization loop, while decentralized planning methods do not ground language instructions or corrections.The passage discusses centralized search and decentralized collision avoidance.Source: 3) Multi-Agent Path Planning · p. 2

4) Language-guided Plan Editing:

CaPE treats language as guidance for editing candidate paths rather than generating paths directly. A joint planner supplies candidate and predicted paths, a VLM synthesizes edits from multimodal context, and a verifier checks them against constraints.

4) Language-guided Plan Editing:

  • CaPE’s path edits are described as human-interpretable and planner-verifiable for collision freedom, addressing limitations in earlier language-guided editing methods.The cited approaches include waypoint prediction [58] [21], cost functions [39], constraints, trajectory modifications [56] [7] [6] [30], control-space changes [10], and policy improvement [41].Source: 4) Language-guided Plan Editing · p. 2

1) Overview:

  • For agent i, a joint planner predicts agent j’s path and proposes candidate paths for i; a VLM uses those paths and agent j’s communication to edit one candidate, which a verifier checks before execution.CaPE can be applied to each agent in a decentralized way and supports human-robot or multi-robot cooperation with varying agent counts.Source: 1) Overview · pp. 2–3, Figure 2 · p. 3

2) Joint Planner:

  • The joint planner generates collision-free candidate routes in different homotopy classes and predicts other agents’ paths from their goals.Different homotopy classes represent qualitatively distinct routes around obstacles; predicted paths provide context for avoiding conflicts with other agents.Source: 2) Joint Planner · p. 3

3) Path Editing Program Synthesis via a VLM:

  • Conditioned on a map, candidate and predicted paths, and communication, the VLM selects a fitting path and edits waypoints or waypoint dwell times through a compact DSL.The defined operations include adding or modifying waypoints and adjusting how long the agent stays at each waypoint.Source: 3) Path Editing Program Synthesis via… · pp. 3–4

4) Training a small VLM as the program synthesizer.:

CaPE’s small VLM is trained on synthetic path-editing examples, and its verifier checks edits against physical constraints and other agents’ predicted paths. The paper describes integrations for multi-robot and human-robot tasks, including settings designed to test generalization beyond training data.

4) Training a small VLM as the program synthesizer.:

  • The small program-synthesizing VLM is trained on 50k synthetic examples covering diverse maps and instructions, but training scenarios contain only two agents.A language model diversifies instruction styles, and each example pairs a map and paths with an instruction and ground-truth program.Source: 4) Training a small VLM as the program… · p. 4

5) Path Verifier:

  • Before execution, the verifier checks each edit against physical constraints and other agents’ predicted paths, rejecting edits that would violate them.Because DSL operations are standalone path modifications, the verifier can check them line by line; a revised program may deviate from the communicated intent.Source: 5) Path Verifier · p. 4

1) Multi-Robot Path Coordination:

  • In multi-robot coordination, agents negotiate safe paths, infer other agents’ goals, then edit their own paths to reflect the negotiation.The integration is described for agents that negotiate when they get close and then safely reach their intended goals.Source: 1) Multi-Robot Path Coordination · p. 4

2) Human-Robot Teaming for Household Object Rearrangement:

  • In household rearrangement, task assignments and human instructions inform the robot’s next goal and path while it completes its assigned tasks.The robot observes the environment and tracks task progress; it can infer the human’s next goal from instructions and state.Source: 2) Human-Robot Teaming for Household… · p. 4

3) Human-Robot Teaming for Joint Carrying:

  • For joint carrying, the human and robot share a goal, and the robot uses human instructions to plan a path for carrying the object in the human’s preferred manner.The human provides instructions about specific ways to reach the goal location.Source: 3) Human-Robot Teaming for Joint… · p. 4

IV. EXPERIMENTS

  • The evaluation domains test distinct challenges: collision avoidance without known intentions, navigation through dense household obstacles, and motion alignment during joint carrying.These settings are out of distribution from training through outdoor objects and multiple agents, denser and different household layouts, and real instructions unlike templated training instructions.Source: §IV · pp. 4–5

A. Experiments 1: Multi-Robot Path Coordination for Autonomous Vehicles

CaPE was evaluated in multi-robot parking coordination and household teaming, where agents had to adapt paths amid obstacles and other agents. Across the reported tasks, CaPE variants improved on baselines, including in three-robot and household scenarios.

1) Setup:

  • The parking evaluation tested two- and three-car navigation without prior knowledge of other cars’ goals or plans, requiring communication and path edits to avoid collisions.Two-car trials began with cars close together; three-car start and goal locations were randomly sampled.Source: 1) Setup · p. 5

2) Baselines:

  • Baselines included planner-only, VLM action and waypoint methods, and RoboPoint; evaluation tracked success rate, inference and token cost, and Success Efficiency Length.Success rate counted collisions as failures, while Success Efficiency Length compared optimal completion time with actual time.Source: 2) Baselines · p. 5

3) Results:

  • In 3-robot trials, CaPE reached 60% success while VLM baselines failed all tasks; in 2-robot trials, CaPE variants significantly improved over baselines.The best 2-robot VLM waypoint baseline achieved 16.67% success, and planner-only succeeded in 23.33% of trials.Source: 3) Results · pp. 5–6, Table III · p. 6, Table II · p. 6

2) Results:

  • In household environments, Gemini-3-pro with CaPE succeeded in 80% of tasks, while VLM agents failed all tasks and waypoint success reached at most 8.33%.The fine-tuned Qwen 2.5 7B model improved over its base model and was comparable to the larger 72B model.Source: Table IV · p. 6, 2) Results · p. 6

1) Setup:

  • Household trials used 20 rearrangement scenarios across seven apartments, with agents completing task-level planning before coordinating paths in multi-room settings.The simulated human followed a pre-planned path and instructed the robot when approaching it.Source: 1) Setup · p. 6

C. Experiment 3: Real-World Human-Robot Joint Carrying

In real-world joint carrying, CaPE was tested with a human and a Stretch 3 robot navigating obstacle-filled layouts. Both CaPE variants outperformed the reported baselines in task success, while analysis describes safer path edits and interpretable coordination behaviors.

1) Setup:

  • Ten participants tested the Stretch 3 robot carrying a PVC pipe through a 6 m × 6 m space with 4–7 obstacles across four maps and four methods.The study randomized model and map order; participants also rated and ranked collaboration after each task.Source: 1) Setup · p. 7, Figure 4 · p. 7

2) Result:

  • CaPE with Gemini-3 outperformed planner-only in participants’ comprehension, safety, and collaboration ranking; the trained model’s gains over the planner were smaller.Participants rated instruction comprehension, movement comfort, efficiency, responsiveness, and safety, and ranked the four models.Source: 2) Result · p. 7, Figure 5 · p. 7
  • Both CaPE variants succeeded in 7 of 10 real-world trials, versus 20% for planner-only and 10% for waypoint generation; the fine-tuned model inferred twice as fast as Gemini-based CaPE.Gemini waypoint generation failed almost all trials.Source: Table V · p. 7, 2) Result · p. 7

A. CaPE provides safe and interpretable language-based plan edit

  • Simulation edits avoided obstacle and boundary collisions, but real-world perception and execution noise could still cause failures; increased obstacle clearance made paths more robust to that noise.Both CaPE and the planner generated physically valid paths computationally, though execution could fail under localization and control noise.Source: A. CaPE provides safe and… · pp. 7–8
  • Gemini-3 often increased obstacle clearance from high-level guidance in 8 of 10 participants, while the trained model applied fine-grained local instructions correctly in 6 of 10 trials.The trained model’s examples include instructions such as moving backward a little.Source: A. CaPE provides safe and… · p. 8
  • Examples show CaPE selecting and refining a planner candidate for clearance, waiting for a human, or moving away from a doorway; explicit DSL edits expose the coordination rationale.The wait operation corresponds to pausing so the human can pass first, and changing the first waypoint moves the robot out of the doorway.Source: A. CaPE provides safe and… · p. 8, Figure 6 · p. 8

B. CaPE generalizes well to diverse scenarios

Across out-of-distribution indoor, outdoor, and real-world tasks, CaPE performs strongly across layouts and coordination scenarios, though the trained model transfers less well to three-agent and indoor tasks. The paper also identifies perception, prediction, safety-fidelity, and scope boundaries, and concludes that CaPE supports safe, interpretable communication-guided path editing.

B. CaPE generalizes well to diverse scenarios

  • CaPE performs strongly across out-of-distribution layouts and coordination scenarios, but the trained model generalizes less well to three-agent and indoor tasks; real-world users’ diverse instructions are understood by both tested models.The trained model performs best in two-robot coordination; outdoor, indoor, and real-world evaluations span different layouts and obstacles.Source: B. CaPE generalizes well to diverse… · p. 8

C. Limitations and future work

  • Perception noise and partial observability may reduce path-edit quality because CaPE assumes reasonably reliable visual perception.The authors suggest more robust perception modules or uncertainty-aware representations as possible improvements.Source: C. Limitations and future work · p. 8
  • Errors in predicted goals or trajectories can produce suboptimal edits, especially when human intentions evolve, while further communication and plan updates can help refine behavior.Source: C. Limitations and future work · p. 8
  • Planner-based safety checks can reject infeasible edits, so the resulting plan may not exactly match human intent—a trade-off between safety enforcement and faithful execution.Source: C. Limitations and future work · pp. 8–9
  • CaPE targets individual-agent adaptation through communication, not learned collective policies for large groups; future work could add gestures or implicit signals and improve robustness to perception uncertainty.Source: C. Limitations and future work · p. 9

VI. CONCLUSION

  • CaPE uses VLM-synthesized path-editing programs checked against physical constraints to adapt agents’ behavior while maintaining geometric safety, and experiments show improved cooperation over strong baselines with safety and interpretability preserved.Source: §VI · p. 9

VII. APPENDIX

The appendix reports two planner ablations and gives the VirtualHome prompt’s inputs, editing tools, and output constraints.

1) Ablation 1: CaPE using single path planner:

  • Using a single-path planner lowers performance in both multi-robot coordination and human-robot teaming, with a steeper drop in human-robot teaming.The authors suggest this may reflect the more spacious carpark map compared with VirtualHome’s indoor environments.Source: Table VI · p. 12, 1) Ablation 1: CaPE using single path… · p. 12

2) Ablation 2: CaPE without safety check:

1) Prompts for VirtualHome Evaluation:

2) Prompts for SimWorld Evaluation:

The SimWorld prompt supplies an instruction, candidate paths, and other agents as reference for generating DSL edits. The real-world prompt frames joint carrying and constrains the robot’s path edits to preserve safe, collision-free movement.

2) Prompts for SimWorld Evaluation:

  • The SimWorld prompt asks for DSL code from an instruction, the target agent’s candidate paths, and other agents’ paths as reference only.It requires Python code with one operation per line and edits only to the target agent.Source: 2) Prompts for SimWorld Evaluation · p. 13
  • SimWorld DSL operations select a path, wait at a waypoint, modify one waypoint’s translation or rotation, or insert a waypoint.Path indices are 0-based, and the target agent name must be quoted.Source: 2) Prompts for SimWorld Evaluation · p. 13

3) Prompts for Real World Evaluation:

  • The real-world prompt presents a robot and human connected by a stick, with the human following the robot and the robot’s rotation setting stick orientation.The prompt describes zero degrees as horizontal with the robot on the left, and clockwise rotations increase theta.Source: 3) Prompts for Real World Evaluation · p. 13
  • The real-world DSL supports selecting either agent’s path and editing waypoints through translation, rotation, insertion, or waiting.Translation offsets are in pixels; inserted waypoints are placed after a specified step.Source: 3) Prompts for Real World Evaluation · p. 13
  • The real-world map key identifies candidate paths by color and homotopy class, obstacles and unreachable regions, start and goal positions, and waypoint indices.The coordinate origin is top-left; +x points right, +y down, and theta 0° faces right.Source: 3) Prompts for Real World Evaluation · p. 13
  • The real-world prompt requires selecting a robot path first, editing only that path, reasoning with obstacle names, and keeping it clear of obstacles and unreachable regions.Original paths are described as collision-free, and edits should try to retain safe distance from obstacles.Source: 3) Prompts for Real World Evaluation · p. 13
  • The real-world prompt says to leave the first and last waypoints unchanged as the start and goal, and return only DSL code.Source: 3) Prompts for Real World Evaluation · p. 13

C. Experiment Details

The experiment-details section describes implementation settings for VirtualHome teaming, SimWorld cooperation, and real-world joint carrying. In VirtualHome, simulated-human instructions and goal inference are prompted from agent positions, paths, instructions, and map state.

C. Experiment Details

  • The experiment-details section covers implementation for VirtualHome human-robot teaming, SimWorld multi-agent cooperation, and real-world joint carrying.Source: C. Experiment Details · p. 14

1) VirtualHome Details:

  • VirtualHome uses Gemini 2.5 Pro to generate simulated-human instructions when human-robot distance falls below d = 5.0, followed by a cooldown of n = 3 execution steps.Goal inference also uses Gemini-2.5-Pro, given candidate receptacles, the human instruction, and map state; both agents are assumed to know the full map and object locations correctly.Source: 1) VirtualHome Details · p. 14
  • The instruction-generation prompt asks a simulated human to give a brief instruction to coordinate movement and avoid collisions, based on the displayed situation and paths.It directs the human to ask the robot to wait or move aside in specified conflicts, and otherwise let it proceed.Source: 1) VirtualHome Details · p. 14
  • The instruction prompt’s map key distinguishes obstacles, unreachable regions, the human’s remaining path and position, the robot’s position and predicted path, and goals.Source: 1) VirtualHome Details · p. 14
  • The instruction-generation prompt requires a short instruction of one to two sentences and output of the instruction text only.Source: 1) VirtualHome Details · p. 14
  • The goal-inference prompt asks which candidate goal best matches an agent’s trajectory direction, curvature, and progression, considering its recent movement and human instruction.It supplies an image, candidate goals, and the instruction, and requires outputting only the selected goal name.Source: 1) VirtualHome Details · p. 14

2) SimWorld Details:

The SimWorld details describe multi-robot parking-lot coordination experiments and the prompts used to infer destinations, negotiate, and issue collision-avoidance instructions.

2) SimWorld Details:

  • The multi-robot experiments use two-robot challenging-start setups and three-robot randomized-start setups, with proximity-triggered negotiation when cars come within 700 units.The supplied description cuts off after introducing the proximity trigger.Source: 2) SimWorld Details · p. 14
  • During negotiation, robots take turns stating intentions, then a VLM gives each robot an instruction; a three-step cooldown prevents immediate repeat negotiations.Gemini 2.5 Pro is used for goal inference and instruction generation, and the robots are assumed to see the full map and object locations.Source: 2) SimWorld Details · p. 15
  • Goal inference combines instruction clues with trajectory direction and curvature, using the start marker and current-position marker to read movement toward a destination.When the trajectory is short, the prompt says to rely more on the instruction; it also warns against assuming nearby goals.Source: 2) SimWorld Details · pp. 15–16
  • Negotiation utterances communicate relative-direction intentions and possible strategies, while the coordinator gives each robot concise, actionable instructions based on intentions and candidate paths.Robots may modify only their own paths, and instructions can refer to path numbers when appropriate.Source: 2) SimWorld Details · pp. 16–17

3) Real World Experiment Details:

The real-world study tests human-robot joint carrying with a Stretch 3 robot across four maps, using spoken instructions and participant ratings of four models.

3) Real World Experiment Details:

  • The human and Stretch 3 robot carry a PVC pipe through obstacle-cluttered maps; the authors describe this setup as more challenging and diverse than prior single-obstacle joint-carrying work.The comparison is with IC-MPC, whose task involved passing a single central obstacle.Source: 3) Real World Experiment Details · p. 17, Table VIII · p. 17
  • For each spoken instruction, the robot stops, localizes, replans waypoints, and navigates in straight lines with a 10 cm obstacle safety margin; crossing that margin counts as failure.Participants and robot both know the destination location.Source: 3) Real World Experiment Details · p. 17
  • After each of four model trials, participants rate instruction understanding, path efficiency and reasonableness, and safety, then rank the models overall.Ratings use a 1-to-7 scale.Source: 3) Real World Experiment Details · p. 17

D. Training Details

The training details describe synthetic data for finetuning CaPE’s motion-program synthesizer, while the pilot study reports instruction patterns that informed the DSL design.

D. Training Details

  • Training finetunes Qwen-2.5-7B on 40,000 labeled path-edit examples generated across 6,000 maps, including narrow corridors and crossroads.The data covers movements, rotations, path selection, obstacle avoidance, waiting, and backing out; training used four A100 GPUs for 36 hours.Source: D. Training Details · p. 17

E. Pilot Study

  • A 10-person pilot study observed both global directives and local edits in joint carrying, motivating a DSL that combines operations to follow participants’ instructions.Participants first carried a long PVC pipe together, then completed a human-robot trial; the study received IRB approval.Source: E. Pilot Study · pp. 17–18
Loading 2602.19304v1…