Source-linked AI summary
CoPlanVLM: Coordinated Path Planning for a Robot Team using Vision Language Models
David R. Martin, Jared Paquet, Mohammad Abdullah Al Faruque, Pramod Khargonekar, Kyle Volle, J. Humberto Ramos, Scott Nivison
TL;DR
VLMs offer a natural-language interface for robot teams, but weak spatial reasoning limits their path planning. CoPlanVLM combines image overlays, task-specific planner and prompt selection, and structured reasoning to plan coordinated paths for two robots. It achieves an 84.4% average success rate versus 50.0% for the closest baseline, with paths 27% shorter when both methods succeed.
Problem
VLMs’ poor spatial reasoning limits their ability to generate long-horizon, collision-free paths for team instructions.
Source: §I · p. 1Method
CoPlanVLM combines image marking, instruction-based planner and prompt selection, and task-specific reasoning to plan coordinated paths for two robots.
Source: §I · pp. 1–2Results
CoPlanVLM achieved an 84.4% overall success rate versus CoNVOI’s 50.0%, and successful paths were 27% shorter in makespan when both methods succeeded.
Source: D. Comparison to Existing Methods · p. 7Takeaways & Limitations
The approach supports coordinated tasks, divided work for efficiency, and detailed maneuvers such as specific routes and loops.
Source: §I · p. 1CoPlanVLM assumes a top-down workspace view, access to a cloud-hosted VLM, and was evaluated on only two robots; its reliability remains insufficient for mission-critical deployment.
Source: §VI · p. 8
Abstract
from arXiv · showhide
We present CoPlanVLM, which uses a Vision Language Model (VLM) to translate a single natural-language instruction and image into coordinated paths for a team of two robots. While VLMs offer a flexible, natural-language interface for commanding robot teams, their weak spatial reasoning has greatly limited VLM-based path planning. CoPlanVLM improves the VLM's spatial reasoning and path planning abilities through three components: (1) an image overlay marking both free space and obstacles, (2) task classification that selects the most appropriate low-level planner and reasoning prompt for each instruction, and (3) task-specific chain-of-thought reasoning that prompts the VLM to identify key objects, allocate roles, and verify routes are collision-free before committing to a plan. Together, these components enable the team to complete complex coordinated maneuvers requiring both robots and divide large tasks between robots to increase efficiency. CoPlanVLM also supports detailed maneuvers involving specific routes, loops, and coverage of large areas, as well as tasks requiring directional reasoning such as north/south or left/right. We validate CoPlanVLM in both Gazebo simulation and on physical Turtlebot 4 hardware. CoPlanVLM attains an average success rate of 84.4% compared to 50.0% for the closest baseline. On trials where both methods succeed, CoPlanVLM's paths are 27% shorter on average. Project code is available at https://github.com/AICPS/CoPlanVLM.
I. INTRODUCTION
CoPlanVLM addresses the difficulty of planning collision-free, long-horizon paths for robot teams from natural-language instructions. It combines image marking, task-specific planner selection, and structured VLM reasoning to plan coordinated routes for two robots.
I. INTRODUCTION
- VLMs’ poor spatial reasoning limits their ability to generate long-horizon, collision-free paths, and prior work does not provide long-horizon team paths for complex maneuvers from natural-language instructions.Prior approaches use cost maps or repeated next-action queries, while waypoint methods described here target single robots.Source: §I · p. 1
- CoPlanVLM achieves an average success rate of 84.4%, compared with 50.0% for CoNVOI, and paths are 27% shorter in makespan when both methods succeed.The system is tested on coordinated tasks, efficiency-focused task division, and tasks requiring little coordination; the paper also reports successful deployment on two Turtlebot 4 robots.Source: §I · p. 1
- CoPlanVLM marks free space and detected objects, classifies each instruction to select a reasoning prompt and low-level planner, and uses task-specific reasoning to identify objects, allocate roles, and verify unobstructed waypoints.The selected planners cover point-to-point navigation, area coverage, and route-constrained maneuvers.Source: §I · pp. 1–2
- The illustrated tasks span Nav2Point, Coverage, and Maneuver categories, with Gazebo plans shown above hardware paths and robot start and stop locations marked.The figure shows two tasks from each category.Source: Figure 1 · p. 2
II. RELATED WORK
Related work uses language models for team planning, navigation, and spatial reasoning, but existing approaches are limited in long-horizon coordinated path planning. CoPlanVLM builds on image marking and waypoint planning to address those limits.
A. LLM Task Planning For Robot Teams
- Prior language-model team planning often decomposes missions into subgoals, while VLM exploration methods select the next frontier and single-robot waypoint planning lacks team coordination or path-feasibility mechanisms.CoPlanVLM extends language-model planning to long-horizon coordinated paths for two robots.Source: A. LLM Task Planning For Robot Teams · p. 2
B. Vision Language Navigation
- Repeated VLM queries and VLA next-action policies produce immediate movement decisions rather than long-horizon plans, limiting coordination and complex maneuvers.Cost-map planning supports efficient long-horizon paths but restricts routes to shortest or lowest-cost paths, unlike waypoint selection that can encode loops, patrols, or passing on a specified side.Source: B. Vision Language Navigation · p. 2
C. Spatial Reasoning for VLMs
- VLM spatial errors can produce instruction-inconsistent, inefficient, or obstacle-crossing paths; prior marker-based methods use labeled grids or free-space markers, while CoPlanVLM distinguishes free space and objects of interest.Its pipeline also uses an occupancy map and task-selected planner and prompt.Source: C. Spatial Reasoning for VLMs · p. 3, Figure 2 · p. 3
III. PROBLEM FORMULATION
The problem is to plan coordinated waypoint sequences for two robots sharing a workspace. The joint plan must satisfy the instruction while avoiding obstacles and minimizing the longer robot path.
III. PROBLEM FORMULATION
- CoPlanVLM plans for two Turtlebot 4 robots, Raph and Donnie, operating in a shared workspace.Source: §III · p. 3
- The system uses an overhead RGB image, each robot’s pose, and a natural-language task instruction, then produces an ordered waypoint sequence for each robot.Each robot executes its sequence in order, and the joint plan combines both sequences.Source: §III · p. 3
- The joint plan must accomplish the instruction with consecutive waypoint segments avoiding obstacles, and minimize makespan estimated by the longer total path length.Planned paths may intersect because inter-robot collision avoidance is enforced after planning through a control barrier function.Source: §III · p. 3
IV. METHODS
CoPlanVLM preprocesses an overhead image into an occupancy map and a marked image, then uses a classifier to select a VLM prompt and low-level planner. The map projects segmented obstacles into world coordinates and applies safety-related corrections.
IV. METHODS
- The system combines CLIPSeg image preprocessing, instruction classification, VLM waypoint selection, and low-level planners to produce coordinated paths.The classifier selects the corresponding prompt and planner for the instruction.Source: §IV · p. 3, A. Image Pre-processing · p. 3
2) Occupancy Map Construction:
- CLIPSeg treats pixels with at least 0.50 confidence for “the floor” as traversable and all others as untraversable.Indoor testing motivates the use of a single traversable prompt.Source: 2) Occupancy Map Construction · p. 3
A. Image Pre-processing
- The occupancy map projects CLIPSeg detections into world coordinates and applies obstacle inflation and map-edge buffers.Detections within 65 pixels of the image boundary are discarded; obstacles are inflated by 0.45 m and cells within 0.25 m of a map edge are marked untraversable.Source: 2) Occupancy Map Construction · pp. 3–4, Figure 3 · p. 4
3) Mark Overlay Construction:
The mark overlay gives the executive VLM labeled grid points, free-space and obstacle markings, and named robot locations. A classifier assigns each instruction to a category for which a corresponding prompt and planner are selected.
3) Mark Overlay Construction:
- An 8 × 14 labeled grid encodes image positions, with cardinal directions mapped to image axes to remove directional ambiguity.Free points appear as blue dots and blocked regions as red “x” marks.Source: 3) Mark Overlay Construction · p. 4
- Robot names are placed at their image locations so the VLM can distinguish the two robots.The robots are named Raph and Donnie.Source: 3) Mark Overlay Construction · p. 4
B. Task Classification
- A classifier LLM first assigns each task instruction to one of three categories.The categories guide the selection of task-specific reasoning prompts and low-level planners.Source: B. Task Classification · p. 4
- Nav2Point covers ordered goal locations when destinations matter but the exact path does not.The category description focuses on reaching one or more destinations in order.Source: B. Task Classification · p. 4
- Coverage applies to patrol or sweep tasks, selecting regions for an efficient low-level controller to visit.The planner selects high-level regions rather than a specific path.Source: B. Task Classification · p. 4
- Maneuver applies when the route itself matters, including loops, specified routes, or avoidance, and serves as a catch-all category.This category is also used when a task does not fit Nav2Point or Coverage.Source: B. Task Classification · p. 4
C. Executive VLM Prompting
The executive VLM receives a task-specific prompt and marked overhead image, reasons through task allocation and route construction, then supplies waypoints or regions to low-level planners.
C. Executive VLM Prompting
- Each task category gets a unique executive-VLM prompt, assembled from six components including the instruction, reasoning scaffold, and marked image.The prompt also includes a preamble, map-overlay description, and category-specific controller explanation with output schema and example.Source: C. Executive VLM Prompting · p. 4
D. Chain of Thought Scaffold
- The VLM’s reasoning scaffold grounds robots and objects, allocates tasks, constructs routes, and verifies selected points are unblocked.Maneuver tasks additionally decompose each robot’s assignment into ordered legs.Source: D. Chain of Thought Scaffold · p. 4
E. Low-Level Planners
- Low-level planners convert VLM-selected waypoints into efficient paths that avoid blocked occupancy-map regions.This planning stage follows the executive VLM’s waypoint selection.Source: E. Low-Level Planners · p. 5
1) A* Path Planning:
- For Nav2Point and Maneuver tasks, the VLM outputs ordered waypoints and A* plans collision-free paths through them from each robot’s initial position.A blocked waypoint is projected to the nearest free cell within 1 m when possible; otherwise it is dropped.Source: 1) A* Path Planning · p. 5
- Nav2Point outputs a single goal or a few goals, while Maneuver outputs a longer sequence specifying a precise route.Poor VLM waypoint selections can still lead to task failures.Source: 1) A* Path Planning · p. 5
- When a blocked waypoint cannot be replaced within 1 m, the planner drops it, so poor VLM selections can still cause task failures.This repair attempts to preserve VLM intent while maintaining safety.Source: 1) A* Path Planning · p. 5
2) Coverage Path Planning:
Coverage planning selects regions and computes an efficient visiting order, while path tracking and a control-barrier-function filter govern robot motion and collision avoidance.
2) Coverage Path Planning:
- Coverage assigns each robot unordered regions, requires visits to every marked point within them, and computes an efficient open-path visiting order.The order is found with nearest-neighbor construction refined by 2-opt; edge costs are Dijkstra shortest-path distances on the inflated occupancy grid.Source: 2) Coverage Path Planning · p. 5
1) Path Tracking:
- A 10 Hz proportional controller steers toward the active goal, turns in place beyond a 50° heading error, and advances goals within 0.1 m.Forward speed is proportional to remaining distance.Source: 1) Path Tracking · p. 5
2) Safety filter.:
- A control-barrier-function quadratic program filters nominal commands using barriers for the peer robot and nearest occupied map cell.The barriers use a look-ahead point 0.10 m ahead of the robot center.Source: 2) Safety filter · p. 5
V. EXPERIMENTS
CoPlanVLM was evaluated with two Turtlebot 4 robots in simulation and on physical hardware. The 18 prompts tested different coordination demands, while offline comparisons assessed whether planned paths completed instructions and avoided obstacles.
V. EXPERIMENTS
- Two Turtlebot 4 robots were evaluated in a simulated warehouse and in real-world experiments at the University of Florida Autonomous Vehicle Laboratory.GPT-4o served as CoPlanVLM’s classifier LLM and executive VLM, and as the VLM for all baselines.Source: §V · p. 5
A. Tasks
- 18 prompts covered Nav2Point, Coverage, and Maneuver tasks, with half of each category tested in simulation and half at the University of Florida AVL.Tasks included cases requiring both robots, cases where dividing work could improve efficiency, and cases requiring little coordination; several also tested directional reasoning.Source: A. Tasks · p. 5
B. Offline Testing
- Offline baseline comparisons and ablations evaluated planned paths directly for task completion and obstacle avoidance, without executing them on robots.Each of the 18 prompts was run ten times per method; results were reported in Table I and Figure 4.Source: B. Offline Testing · p. 5
C. Performance Metrics
The evaluation measures task completion and path efficiency, then compares CoPlanVLM with adapted VLM-based planning approaches. CoPlanVLM had the highest overall success rate, and its successful paths were shorter than those from CoNVOI marking.
C. Performance Metrics
- Success Rate measures task completion without obstacle intersection; Coverage tasks additionally require at least 95% coverage of grid points in the described area.Table I reports CoPlanVLM success rates and average makespan for all evaluation prompts, with 10 trials per prompt.Source: C. Performance Metrics · p. 5, Table I · p. 6
- Normalized Makespan compares mean path makespan by task against CoPlanVLM; values above 1.0 indicate longer paths and values below 1.0 shorter paths.Makespan is the longer of the two robots’ path lengths; averages include successful trials only, and a baseline task with no successful trials is excluded.Source: C. Performance Metrics · pp. 5–6
D. Comparison to Existing Methods
- The comparison used three adapted VLM-based path-planning approaches because prior work had not investigated coordinated path planning from natural language for these complex behaviors.Grid Overlay, CoNVOI Marking, and Pixel Selection were adapted to the two-robot setting.Source: D. Comparison to Existing Methods · pp. 6–7
- 84.4% overall success rate for CoPlanVLM exceeded CoNVOI’s 50.0%, while successful paths had 27% shorter makespan than CoNVOI marking (1.00 vs. 1.37).CoPlanVLM also outperformed baselines on Coverage, Nav2Point, and Maneuver tasks; the authors attribute its shorter paths largely to assigning subtasks to suitable robots and dividing large coverage tasks.Source: Table II · p. 7, D. Comparison to Existing Methods · p. 7
E. Ablation Testing
Ablations examined obstacle marking, planner selection, and chain-of-thought reasoning. Removing components reduced success on several task types, though obstacle marking’s effect differed for Coverage tasks.
E. Ablation Testing
- Without obstacle marking, Nav2Point and Maneuver success rates fell by −40% and −18.3%, while Coverage performance improved slightly by +15%.The authors hypothesize that the coverage planner’s removal of unsafe points can make obstacle labels noisy for Coverage tasks.Source: E. Ablation Testing · p. 7
- Using only the Maneuver planner reduced success rate by −13.3% on Nav2Point and −55% on Coverage tasks, with higher makespan.The authors conclude that relying on one planner for all tasks reduces solution quality.Source: E. Ablation Testing · p. 7
- Removing CoT reasoning reduced average success rate by −35.5% over all tasks, the greatest overall ablation effect.Table III reports the ablation study averaged by task category.Source: Table III · p. 7, E. Ablation Testing · p. 7
F. Robot Deployment
Hardware deployments produced success rates similar to offline testing; the conclusion identifies remaining reliability and deployment boundaries, including obstacle detection and the two-robot evaluation scope.
F. Robot Deployment
- 83.3% vs. 81.1% success rates indicate similar performance in hardware deployment and offline testing.The hardware evaluation used two Turtlebot 4 robots and fresh overhead images with newly generated plans for each trial.Source: F. Robot Deployment · p. 7
VI. CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
- CoPlanVLM combines image preprocessing, task categorization, and structured chain-of-thought reasoning to improve spatial reasoning, coordination, and path planning over baselines.The conclusion characterizes the resulting paths as more efficient and reports improved success rates compared with baselines.Source: §VI · p. 8
- The method assumes a top-down workspace view and cloud-hosted VLM, was evaluated on only two robots, and remains insufficiently reliable for mission-critical deployment.Missed small-obstacle detections contributed to failures; the authors suggest complementary sensing such as LiDAR to refine the occupancy map.Source: §VI · p. 8