PaperlayerView exact revision on arXiv

Source-linked AI summary

SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models

Shyam Sundar Kannan, Vishnunandan L. N. Venkatesh, Byung-Cheol Min

arXiv:2309.10062v2cs.RO

TL;DR

Multi-robot task planning must handle diverse, complex, and ambiguously described tasks beyond fixed algorithms. SMART-LLM uses LLM-guided decomposition, coalition formation, and skill-based allocation, achieving favorable outcomes across task categories and LLM backbones.

  • Problem

    Fixed algorithms struggle to transition across diverse, complex, and ambiguously described natural-language tasks in multi-robot planning.

  • Method

    SMART-LLM uses environment and robot-skill information with few-shot programmatic LLM prompting for task decomposition, coalition formation, and skill-based assignment.

  • Results

    SMART-LLM delivers favorable outcomes across task categories and LLM backbones, with Claude-3 reaching TCR 1.0 and SR 0.87 on simple tasks.

  • Takeaways & Limitations

    The approach handles task instructions of varying complexity and supports transitions from simulation-generated plans to real robot systems.

Abstract

from arXiv · show

In this work, we introduce SMART-LLM, an innovative framework designed for embodied multi-robot task planning. SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models (LLMs), harnesses the power of LLMs to convert high-level task instructions provided as input into a multi-robot task plan. It accomplishes this by executing a series of stages, including task decomposition, coalition formation, and task allocation, all guided by programmatic LLM prompts within the few-shot prompting paradigm. We create a benchmark dataset designed for validating the multi-robot task planning problem, encompassing four distinct categories of high-level instructions that vary in task complexity. Our evaluation experiments span both simulation and real-world scenarios, demonstrating that the proposed model can achieve promising results for generating multi-robot task plans. The experimental videos, code, and datasets from the work can be found at https://sites.google.com/view/smart-llm/.

I. INTRODUCTION · II. RELATED WORKS

SMART-LLM addresses the difficulty of planning heterogeneous multi-robot tasks expressed in natural language by using LLMs for decomposition, coalition formation, and skill-based allocation. The paper contributes a benchmark and evaluates the framework in simulation and real-world settings, while positioning it against task-specific and incomplete prior methods.

  • I. INTRODUCTION: Multi-robot systems improve efficiency, scalability, and adaptability but become more complex when robots differ in type and skill level.The paper motivates leveraging extensive prior knowledge to interpret natural-language instructions for efficient planning.
  • I. INTRODUCTION: Traditional fixed-algorithm planners struggle to adapt across diverse tasks and complex environments, especially when instructions are imprecise or incomplete.Transitioning between tasks can require substantial code modification, increasing the challenge of natural-language task planning.
  • I. INTRODUCTION: SMART-LLM uses LLMs with Python programming scripts to convert high-level instructions into coordinated multi-robot task plans.The framework targets embodied agents and supports task decomposition, coalition formation, and skill-based task assignment.
  • I. INTRODUCTION: The paper contributes a benchmark dataset spanning elemental to complex multi-agent tasks in the AI2-THOR simulation platform.The dataset is designed to evaluate multi-agent task-planning systems across a spectrum of task complexity.
  • I. INTRODUCTION: SMART-LLM is implemented and tested across a wide range of tasks in both simulated and real-world settings.This evaluation is presented as a main contribution alongside the planning framework and benchmark dataset.
  • II. RELATED WORKS: Multi-robot task planning commonly comprises task decomposition, coalition formation, task allocation, and task execution, with prior allocation methods including negotiation, auctioning, consensus, and reinforcement learning.Existing approaches are often tailored and optimized for specific goals or applications.
  • II. RELATED WORKS: Prior methods often automate only subsets of planning, require task-specific demonstrations, or remain limited to particular tasks; SMART-LLM performs all four steps and uses few-shot prompting to generalize across tasks.The related work also notes that LLM-based multi-agent approaches improve cooperation but do not specifically address task-plan creation for larger teams.

III. PROBLEM FORMULATION

SMART-LLM formulates multi-robot planning as converting a high-level instruction into executable steps in an environment, while maximizing robot utilization through feasible parallel execution. The formulation models heterogeneous robots with constrained skill sets and supports decomposition, coalition formation, and allocation.

  • Planning Objective: The planning goal is to interpret instruction I, compute completion steps, and produce an executable task plan that parallelizes tasks when feasible.Execution occurs in an environment E containing entities and objects.
  • Robot and Skill Model: The system operates with N heterogeneous embodied robots, each having a skill set S_n ⊆ ∆ subject to task-specific constraints.Skills may be pre-implemented or accessible through APIs; constraints can include limits such as maximum pickup mass.
  • Planning Stages: The task-planning pipeline identifies subtasks, matches them to robot skills and properties, forms teams for skill gaps, and assigns subtasks to robots or teams.The formulation explicitly includes task decomposition and coalition formation as planning stages.
  • Execution Structure: The resulting plan can execute independent subtasks concurrently and defer dependent subtasks until their prerequisites finish.The examples parallelize two subtasks before executing a subsequent task.

LLM LLM

SMART-LLM decomposes an instruction into temporally ordered subtasks using robot skills and the environment, then forms robot coalitions when individual robots cannot satisfy a subtask’s skill requirements. Its system overview frames this process as staged LLM-guided planning, including task decomposition and coalition formation.

  • System Overview: SMART-LLM organizes planning into stages including task decomposition and coalition formation, using prompts with robot skills, objects, examples, and coalition policies.The system overview describes task decomposition prompts combined with the input instruction and coalition-formation prompts describing robot teams for decomposed tasks.
  • Task Decomposition: The instruction is decomposed into a temporarily ordered set of K subtasks based on robot skills and the environment.Subtasks may share the same temporal precedence and therefore execute in parallel.
  • Task Allocation: A subtask can be allocated to one robot when that robot’s skills contain all skills required by the subtask.The allocation condition is expressed as the required skill set being a subset of the robot’s available skills.
  • Coalition Formation: When no single robot satisfies the skill constraint, SMART-LLM forms a team of two or more robots to perform the subtask.The coalition is represented as a team of Q robots, with each robot contributing its own skills.

IV. METHODOLOGY

The approach uses LLMs for task decomposition, coalition formation, and task allocation in multi-robot planning. Pythonic prompts guide executable code generation through structured syntax, commented examples, and compact dictionary representations of robot skills and object properties.

  • LLM-based task planning: The method applies LLMs to task decomposition, coalition formation, and task allocation for multi-robot task planning.
  • Prompt design: Pythonic prompts are preferred because they support direct executable-code generation and provide structured syntax that improves LLM comprehension.
  • Prompt design: Prompt samples use line-by-line and block comments to summarize tasks and clarify execution and allocation requirements.
  • Prompt design: Robot skills and object properties are encoded as Python dictionaries, offering concise representations that reduce token size and aid LLM comprehension [39].

A. Stage 1: Task Decomposition

Stage 1 decomposes an input instruction into independent sub-tasks with action sequences, using environment information, robot skills, and few-shot examples to generate an executable plan.

  • Stage 1: Task Decomposition: The LLM generates independent sub-tasks T and action sequences for input instruction I using environment details E and primitive robot skills ∆.These inputs constrain decomposition so the resulting tasks can be performed in the specified environment.
  • Stage 1: Task Decomposition: Few-shot prompting supplies robot-skill and environment information, sample tasks, and Python code-based decomposed plans before the LLM processes the input task.The prompted LLM combines these materials with instruction I to generate the decomposed sub-tasks T.

B. Stage 2: Coalition Formation

Coalition formation assigns suitable robot teams to decomposed sub-tasks by matching required skills with individual robot skills and capabilities. The LLM uses few-shot coalition policies to handle single-robot assignments, collaboration, and capability constraints.

  • B. Stage 2: Coalition Formation: The stage forms robot teams for each decomposed sub-task by analyzing required skills alongside individual robots’ skills and capabilities.
  • B. Stage 2: Coalition Formation: The LLM receives decomposed-task examples, coalition policies, and available robots with their skills to generate an assignment policy for the input tasks.
  • B. Stage 2: Coalition Formation: Coalition policies cover one-to-one assignments when one robot has all required skills and multi-robot collaboration when skills are distributed across robots.
  • B. Stage 2: Coalition Formation: Additional robots address capability limitations even when one robot has the necessary skills, such as a maximum weight constraint during pickup.

C. Stage 3: Task Allocation

Stage 3 allocates individual robots or robot teams to decomposed sub-tasks according to the coalition formation policy. It uses decomposed task samples, coalition policies, available robot skills, and allocated plans to generate the final executable code.

  • C. Stage 3: Task Allocation: Task allocation assigns a specific robot or robot team to each sub-task according to the preceding coalition formation policy.The Stage 3 prompt incorporates decomposed sub-tasks and the coalition formation policy for the input task.
  • C. Stage 3: Task Allocation: The Stage 3 prompt includes decomposed plans, available robots and skills, coalition policies, and allocated plans before producing executable code in Stage 4.

D. Stage 4: Task Execution · V. EXPERIMENTS · A. Benchmark Dataset

SMART-LLM executes allocated multi-robot plans through an interpreter that calls robots’ low-level skills, using virtual or physical teams. Its benchmark uses AI2-THOR household environments and spans four task categories with increasing coordination and heterogeneity requirements.

  • D. Stage 4: Task Execution: Allocated task plans are executed by an interpreter that invokes robots’ low-level skills for virtual or physical multi-robot teams.Stage 4 illustrates execution for a household instruction involving lights and television.
  • D. Stage 4: Task Execution: The execution stage shows robot-view sequences with captions identifying ongoing steps and green-marked actions that have been completed.
  • V. EXPERIMENTS · A. Benchmark Dataset: The experiments use a benchmark for natural-language multi-robot task planning, built from deterministic AI2-THOR household environments, actions, instructions, and floor plans.The dataset contains 36 high-level instructions.
  • V. EXPERIMENTS · A. Benchmark Dataset: Each benchmark task includes final ground-truth object states and symbolic goal conditions, including positions and conditions such as heated, cooked, sliced, or washed.
  • V. EXPERIMENTS · A. Benchmark Dataset: The dataset organizes tasks into four categories to evaluate SMART-LLM across diverse task complexities.
  • V. EXPERIMENTS · A. Benchmark Dataset: Elemental tasks use one robot, whereas Simple tasks involve multiple objects and sequential-or-parallel decomposition with robots assumed to share all necessary skills.
  • V. EXPERIMENTS · A. Benchmark Dataset: Compound tasks permit sequential, parallel, or hybrid execution with heterogeneous specialized robots, while Complex tasks require strategic team assignments because no robot can independently perform every sub-task.Complex tasks combine decomposition, multi-robot engagement, multiple objects, and heterogeneous teams.

B. Simulation Experiments · C. Real-Robot Experiments · D. Evaluation Metrics

The experiments evaluate SMART-LLM on unseen AI2-THOR benchmark tasks, across multiple language-model backbones and real aerial-ground robot settings. Performance is assessed using five metrics comparing achieved execution states with dataset ground truth.

  • B. Simulation Experiments: SMART-LLM is evaluated in AI2-THOR using benchmark tasks spanning parallelizable, sequential, and other task-planning scenarios, with separate prompts for decomposition, coalition formation, and allocation.The setup uses 5 decomposition examples, 3 coalition-formation examples, and 4 task-allocation examples.
  • B. Simulation Experiments: All benchmark tasks are unseen during testing, and SMART-LLM is evaluated with GPT-4, GPT-3.5, Llama-2-70B [7], and Claude-3-Opus.The prompts use different tasks and AI2-THOR floorplans from those in the dataset.
  • C. Real-Robot Experiments: Real-robot experiments test visibility coverage and object-image capture using heterogeneous aerial and ground robots with different skills and visibility capabilities.SMART-LLM generates plans for regions of varying sizes and objects requiring image capture.
  • B. Simulation Experiments: Table I evaluates SMART-LLM and baseline approaches in the AI2-THOR simulator across benchmark task categories.The supplied passage identifies the table’s scope but does not provide its cell values or establish a comparative winner.
  • C. Real-Robot Experiments: The real-robot task plans use GoToLocation, ClickPicture, and Patrol skills, while reusing prompt samples grounded in the AI2-THOR simulation experiments.These skills are described as essential for proficient task execution.
  • D. Evaluation Metrics: Evaluation uses Success Rate, Task Completion Rate, Goal Condition Recall, Robot Utilization, and Executability, comparing achieved post-execution states with dataset ground-truth final states.These metrics assess task success, completion, goal satisfaction, team efficiency, and action executability.
  • D. Evaluation Metrics: GCR measures achieved goal conditions, TCR equals 1 exactly when GCR equals 1, RU compares transition counts with ground truth, and Exe measures executable actions.SR equals 1 only when both GCR and RU equal 1; otherwise, it is 0.

VI. RESULTS AND DISCUSSION · A. Simulation Experiments

Simulation experiments show that SMART-LLM generally produces favorable multi-robot plans across LLM backbones and task complexities, while performance depends on decomposition, sequencing, and allocation quality. Additional tests indicate that the method can recognize infeasible scenarios, and ablations show that comments and coalition formation contribute to success.

  • A. Simulation Experiments: SMART-LLM delivers favorable outcomes across LLM backbones on unseen tasks spanning elemental, simple, compound, and complex categories.The results are summarized as averages across dataset categories and include comparisons with baseline methods.
  • A. Simulation Experiments: GPT-3.5 sometimes struggles to decompose elemental tasks, whereas accurate decomposition enables SMART-LLM to assign robots appropriately.This result indicates that backbone reasoning ability affects task-planning reliability.
  • A. Simulation Experiments: Claude-3 achieves the strongest simple-task performance, while GPT-4 and Claude-3 attain perfect TCR scores of 1.0 but lower SR because robots execute sequentially rather than in parallel.The resulting sequential execution also affects robot utilization.
  • A. Simulation Experiments: Compound and complex tasks achieve a 70% success rate across all tested LLM backbones, although occasional sequencing and robot-team assignment errors remain.Additional prompt samples may mitigate these errors, but token limitations can hinder that optimization; GPT-3.5 underperforms other models in these tasks.
  • A. Simulation Experiments: In infeasible scenarios, GPT-4- and Claude-3-based SMART-LLM correctly detect that no robot has the required skills and refrain from generating an allocation plan.These scenarios are excluded from Table I because no feasible code can be generated for metric measurement.
  • A. Simulation Experiments: Removing comments generally lowers success rates, especially for compound and complex tasks, showing that natural-language instructions with code aid reasoning and logical-structure understanding.Without comments, decomposition and allocation remain similar on simple and elemental tasks but deteriorate on more complex tasks.
  • A. Simulation Experiments: Removing coalition formation also decreases success, indicating that this stage contributes materially to SMART-LLM’s task-planning performance.The supplied ablation results identify coalition formation as an important component of the method.

B. Real-Robot Experiments · VII. CONCLUSIONS AND FUTURE WORK

Real-robot experiments show that the method generates and allocates task plans for unseen visibility-based tasks, while the conclusions emphasize prompt-guided planning, adaptability, and simulation-to-real transfer across complex settings.

  • B. Real-Robot Experiments: The real-robot evaluation included ablation studies summarized in Table III.The supplied table passage identifies the ablation-study scope but provides no individual ablation values or outcomes.
  • B. Real-Robot Experiments: Across real-robot tasks, the method correctly generated task plans and allocated appropriate numbers of robots, including for completely unseen visibility-based tasks.The experiments used regions with different areas and robots with different visibility areas, and executed the task seamlessly on real robots.
  • B. Real-Robot Experiments: For “patrol the regions,” robots were assigned to regions according to visibility area and then patrolled their allocated regions.Figure 3 depicts the robot team, regions, post-planning assignments, and resulting patrol behavior.
  • VII. CONCLUSIONS AND FUTURE WORK: The study targets task-plan generation for heterogeneous robot teams using prompts tailored to four key multi-robot task-planning stages.Each prompt incorporates environmental attributes and individual robot capabilities when generating a task plan.
  • VII. CONCLUSIONS AND FUTURE WORK: Experiments validated that the proposed method handles task instructions with varying complexities.The benchmark and real-robot evaluations support this conclusion across the studied task settings.
  • VII. CONCLUSIONS AND FUTURE WORK: The approach generalizes to new and unexplored environments, robot types, and task scenarios, demonstrating adaptability beyond its sampled settings.The conclusions characterize this generalization as seamless across those three dimensions.
  • VII. CONCLUSIONS AND FUTURE WORK: The method streamlines simulation-to-real deployment by enabling simulation task-plan samples to generate plans for real robot systems.This transfer capability is presented as a bridge from simulated planning to real-world robot applications.
Loading 2309.10062v2…