PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems

Artem Lykov, Maria Dronova, Nikolay Naglov, Mikhail Litvinov, Sergei Satsevich, Artem Bazhenov, Vladimir Berman, Aleksei Shcherbak, Dzmitry Tsetserukou

arXiv:2312.09348v1cs.RO

TL;DR

LLM-MARS addresses limited research on generating complete behavior trees for robots and the need to reconfigure multi-agent systems when tasks change. It uses an LLM to generate behaviors and support dialogue, with robot trials and expert evaluations reporting task and response outcomes.

  • Problem

    Research on LLM generation of complete robot behavior trees that handle many possible scenarios remains limited, while task modifications in existing multi-agent systems can require expert reconfiguration.

    Source: §I · p. 1, §IV · p. 4

  • Method

    LLM-MARS uses a Falcon 7B-based LLM to generate behavior trees for multi-agent robots and support dialogue about task outcomes.

    Source: §VIII · p. 9, §IX · p. 9

  • Results

    79.28% average percentage of correctly added tasks within a command; commands with up to two tasks exceeded 90%, and question responses scored 72.8% accuracy, 4.71 relevance, and 4.89 informativeness.

    Source: §VIII · p. 9

  • Takeaways & Limitations

    The authors identify logistics, autonomous exploration, and collaborative manipulator robots as potential application areas for the approach.

    Source: §IX · p. 10

    The reported trials were conducted within the Eurobot 2023 game rules.

    Source: §IX · p. 9

Abstract

from arXiv · show

This paper introduces LLM-MARS, first technology that utilizes a Large Language Model based Artificial Intelligence for Multi-Agent Robot Systems. LLM-MARS enables dynamic dialogues between humans and robots, allowing the latter to generate behavior based on operator commands and provide informative answers to questions about their actions. LLM-MARS is built on a transformer-based Large Language Model, fine-tuned from the Falcon 7B model. We employ a multimodal approach using LoRa adapters for different tasks. The first LoRa adapter was developed by fine-tuning the base model on examples of Behavior Trees and their corresponding commands. The second LoRa adapter was developed by fine-tuning on question-answering examples. Practical trials on a multi-agent system of two robots within the Eurobot 2023 game rules demonstrate promising results. The robots achieve an average task execution accuracy of 79.28% in compound commands. With commands containing up to two tasks accuracy exceeded 90%. Evaluation confirms the system's answers on operators questions exhibit high accuracy, relevance, and informativeness. LLM-MARS and similar multi-agent robotic systems hold significant potential to revolutionize logistics, enabling autonomous exploration missions and advancing Industry 5.0.

I. INTRODUCTION

The paper addresses limited research on LLM generation of complete robot behavior trees for varied scenarios. It applies a fine-tuned LLM to multi-agent robot control, combining behavior generation with discussion of task outcomes.

I. INTRODUCTION

  • Research on LLMs generating complete robot behavior trees that handle many possible scenarios remains limited.The paper identifies this as a gap in using natural-language techniques to generate robot operating sequences.Source: §I · p. 1
  • The behavior-tree generation model is a fine-tuned variant of Falcon 7B, selected as a proof of concept despite the challenge of keeping up with newer models.The paper says Falcon 7B ranked highly when the system was constructed.Source: §I · p. 1
  • The training data pairs robot commands with behavior trees, including examples generated through the ChatGPT API and manually written trees.Physical-robot experiments were conducted to evaluate the multi-agent system under conditions intended to resemble real-world scenarios.Source: §I · pp. 1–2
  • Eurobot 2023 provides a test setting involving collection, sorting, transport, placement, navigation, and point forecasting by robots.The competition includes a pair of robots working in tandem and responding to adversary actions.Source: §I · p. 2
  • LLM-MARS applies a multimodal LLM to a tandem of mobile robots, with modules for behavior-tree generation and discussion of task outcomes.The strategy-generation process maps user-defined tasks to a behavior tree for robots to solve in their environment.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORKS

Related work describes LLM applications in language-driven robotics and notes limited research on generating robot behavior trees. The paper discusses constraints in prior approaches and the challenge of assembling task-specific training data.

II. RELATED WORKS

  • LLMs have been used to decompose natural-language tasks for robots and support human-robot interaction, alongside work on manipulation and other robotic applications,, [16], [17], [18],,,.Source: §II · p. 2
  • Research on LLM generation of robot behavior trees remains limited [22], [23], and traditional behavior design often requires specialists, [31], [32], [33],,.Source: §II · pp. 2–3
  • A prior approach used an OpenAI model to populate fixed behavior-tree structures restricted to sequence and action nodes, whereas the paper argues task-specific fine-tuning can yield more diverse trees [22].Source: §II · p. 3
  • Examples include PaLM-E [36] and a robotics transformer for real-world control [37], while Spot has been used for vocal reports and navigation but not physical-object interaction.Source: §II · p. 3
  • Building varied behavior-tree training data is difficult because it requires substantial specialist input; the paper uses a self-instruct approach based on text-davinci-003 [40], [41],.Source: §II · p. 3

III. ADVANTAGES OF BEHAVIOR TREES APPROACH

Behavior trees represent robot tasks hierarchically, with leaf nodes executing tasks and branch nodes controlling flow. Their modular structure supports node changes without rebuilding the entire structure.

A. Advantages of Using Behavior Trees in Robotics

  • A behavior tree is a rooted hierarchy whose leaves execute tasks and whose branches define control flow.Action leaves specify primitive tasks; condition leaves evaluate Boolean conditions, while sequence and fallback branches use failure and success signals, respectively.Source: A. Advantages of Using Behavior Trees… · p. 3
  • Behavior trees offer a modular alternative to state machines: nodes can be added, removed, or replaced without reconstructing the whole structure.Source: A. Advantages of Using Behavior Trees… · p. 3
  • Behavior trees have been applied in robotics competitions including the DARPA Robotics Challenge, RoboCup, and Eurobot,,.The cited examples include robot driving and manipulation, multi-robot soccer and search-and-rescue, and Eurobot navigation and object manipulation.Source: A. Advantages of Using Behavior Trees… · p. 3

B. Advantages of Using Behavior Trees as Output of LLM

The paper presents behavior trees as a modular LLM output and describes a multi-agent control approach in which one LLM manages agents through generated behaviors and dialogue. This design targets task changes and complex collaboration among robots.

B. Advantages of Using Behavior Trees as Output of LLM

IV. MULTIMODAL LLM TO MULTIAGENT ROBOT CONTROL

  • Generating complex action sequences is presented as strategically useful, whereas the paper says LLM applications have less significance for repetitive, monotonous tasks.Source: §IV · p. 4
  • The LLM builds agent-specific behavior trees for collaborative tasks; agents execute them autonomously and periodically report environmental information and results.The system supports different numbers and types of robots and generates trees that include checks for multiple conditions and responses.Source: §IV · p. 4
  • The paper argues that assigning independent AI to each robot is overly ambitious when many robots contribute simple components to a broader task.Source: §IV · p. 4
  • The proposed system assigns one LLM-based AI manager high-level control, allowing task modifications through instructions rather than expert-led reconfiguration.The paper contrasts this with existing systems that rely on human operators or task-specific predefined programs and require experts and time for changes.Source: §IV · p. 4

A. Multimodal LLM to Control Eurobot Robots

The paper integrates a multimodal LLM with Eurobot robots whose existing behavior-tree support lets generated strategies run alongside manually scripted ones. Because the robots could not run the 7B-parameter model onboard, queries and responses pass through a remote server.

A. Multimodal LLM to Control Eurobot Robots

  • The Eurobot robots were used to test whether generated behavior trees could support human interaction and run like the robots’ manually scripted strategies.The 2023 competition involved collecting and sorting colored cakes and tallying cherries for transport to designated areas.Source: A. Multimodal LLM to Control Eurobot… · p. 4
  • The robots could not execute the 7B-parameter LLM on their own hardware, so natural-language queries went to a remote server and responses and strategies returned to the robots.The leased server used Intel Broadwell and NVIDIA Tesla V100 hardware.Source: A. Multimodal LLM to Control Eurobot… · p. 4

V. SYSTEM OVERVIEW

  • The system overview presents the interplay of robotic and software components as a guide for understanding and replicating the LLM-driven multi-agent system.Source: §V · pp. 4–5

A. Mechanical Construction Overview

  • Each robot has an omni-wheel platform and suspension, with grippers, a sorting mechanism, and cherry-handling mechanisms attached to its steel frame.The three grippers split cake layers between lower and upper parts; the sorter lifts and rotates, while the cherry mechanisms collect and dispense cherries.Source: A. Mechanical Construction Overview · p. 5

B. Embedded Systems Overview

The embedded electronics connect the high-level computer to onboard devices, while the high-level software combines sensor data, localization, vision, navigation, and behavior trees. Together these modules provide robot positioning, path planning, and strategy execution.

B. Embedded Systems Overview

  • The electronics bridge the high-level computer and onboard devices, host time-critical operations, manage battery power, and center on an STM32F407VG main control board.The high-level computer connects to the board by USB, and the board supplies control and connectivity to downstream electronic entities.Source: B. Embedded Systems Overview · p. 5

C. High-Level Systems Overview

  • The high-level system uses ROS2 Humble on an Intel NUC to exchange commands and responses with the STM32 over USART and publish wheel odometry at 40 Hz.It processes sensor data including LIDAR, camera, and wheel-encoder readings.Source: C. High-Level Systems Overview · pp. 5–6
  • Localization fuses wheel-encoder, LiDAR, and computer-vision measurements in a particle filter to refine the robot’s pose.LiDAR landmarks are used in the filter’s update phase after detection and data association.Source: C. High-Level Systems Overview · p. 6
  • The computer-vision system detects cakes and their colors and layers, and uses ArUco markers to localize robots from an overhead camera view.The paper notes that marker-based localization is useful in dynamic multi-robot settings where LiDAR localization may be inaccurate.Source: C. High-Level Systems Overview · p. 6
  • Navigation builds an occupancy grid, plans paths to behavior-tree targets with RRT*, replans around opponents, and sends smoothed velocities to the embedded controller.It issues a stop command and feedback to the behavior-tree module if a target is unreachable.Source: C. High-Level Systems Overview · p. 6
  • The behavior-tree module provides human-readable strategies as XML, using action and condition nodes with input and output ports.Source: C. High-Level Systems Overview · p. 6

VI. LLM TRAINING METHODS AND RECIPES

The authors fine-tuned Falcon 7B with LoRA to generate behavior trees from operator commands. They built paired command-and-tree data from predefined robot nodes, using SubTree nodes to construct behavior in stages.

VI. LLM TRAINING METHODS AND RECIPES

  • Falcon 7B was fine-tuned with LoRA to construct behavior trees for multi-agent robots from natural-language operator commands.LoRA adjusts low-rank matrices added to transformer layers rather than retraining all model parameters.Source: §VI · p. 6

A. Behavior Tree Generation Module Fine-tuning Process

  • Each training sample pairs a human-readable behavior instruction with a structurally valid behavior tree of action and condition nodes, sometimes including SubTree nodes for staged generation.The authors use SubTree nodes to avoid generating large constructions in one pass, which they say would increase memory needs quadratically with sequence length.Source: A. Behavior Tree Generation Module… · p. 7
  • A Python script generated varied trees from predefined robot nodes, and ChatGPT paraphrased their command descriptions; the resulting dataset contained 7,500 command–tree pairs.Fine-tuning ran for three epochs and 10 hours on an Nvidia Tesla V100.Source: A. Behavior Tree Generation Module… · p. 7

B. Question Answering Module Fine-tuning Process

The authors use one LLM with task-specific LoRA adapters to support behavior-tree generation and question answering without running two 7B models on a robot. The question-answering adapter uses behavior-result XML and user questions as input.

B. Question Answering Module Fine-tuning Process

  • Using one LLM with multiple adapters reduces the need to run two 7B models on a robot, but switching adapters takes about 40 seconds and a shared adapter harmed behavior-tree structure correctness.Source: B. Question Answering Module… · p. 7
  • The architecture combines the LLM with multiple task adapters, and the authors say further adapters could expand the system’s capabilities.Source: Figure 4 · p. 7, B. Question Answering Module… · p. 7
  • The question-answering adapter receives a user question and XML context assembled from the behavior tree and control-sensor data; its training set contained 11,000 generated examples.Training took three epochs and 12 hours on an Nvidia Tesla V100.Source: B. Question Answering Module… · p. 7

A. Experiment

The experiment tested whether participants could distinguish LLM-generated behavior trees (BTs) from human-authored ones. Their results were comparable to chance, with no significant evidence that the trees could be distinguished.

A. Experiment

  • Fifteen participants, including experienced BT users and novices who received instruction, assessed the trees.All participants provided informed consent and were familiarized with BT structure and operation.Source: A. Experiment · pp. 7–8
  • 4.53 correct answers out of 10 placed participants’ ability to distinguish LLM-generated from human-authored BTs near random chance.Fig. 5 depicts the correct-answer distribution.Source: Figure 5 · p. 8, A. Experiment · p. 8
  • Participants compared randomized pairs of similarly functional BTs, one LLM-generated and one manually authored, and identified which was which.The study used 10 trees of each type.Source: A. Experiment · p. 8
  • The t-test found no significant difference from the chance-level mean of 5/10 (mean score = 0.453, p > 0.05).The authors did not reject the null hypothesis that the BTs could not be distinguished.Source: A. Experiment · p. 8

B. Behavior Generation Performance Evaluation

The evaluation measured behavior-tree task integration across command lengths and separately assessed expert ratings of robot-behavior answers. Task integration was strongest for commands with up to two tasks, while experts rated answers highly for relevance and informativeness.

B. Behavior Generation Performance Evaluation

  • 70.44% accuracy in integrating provided robot tasks into BTs; commands with up to two tasks exceeded 90%, and average correctly added tasks per command was 79.28%.The evaluation used 60 commands spanning one to six tasks, with ten examples per task variant.Source: B. Behavior Generation Performance… · p. 8
  • Task-integration accuracy varied with command task quantity in an experiment where the authors assessed BTs generated for designated tasks.Each task variant had ten examples.Source: B. Behavior Generation Performance… · p. 8

C. Expert Evaluation of Question Responses

  • 72.8% average accuracy, 4.71 relevance, and 4.89 informativeness were the expert assessment results for LLM responses about robot behavior.Ten experts assessed 50 responses.Source: C. Expert Evaluation of Question… · pp. 8–9
  • Krippendorff’s alpha was 0.8239 for accuracy, 0.7229 for relevance, and 0.7594 for informativeness.The reported agreement was robust for accuracy and moderate for relevance and informativeness.Source: C. Expert Evaluation of Question… · p. 9

VIII. RESULTS AND DISCUSSION

In the robot trials, task integration accuracy depended on command length, and expert ratings of robot-behavior answers were high. The authors identify handling commands with more tasks as an area for further refinement.

VIII. RESULTS AND DISCUSSION

  • 70.44% of provided robot tasks were integrated into BTs; commands with up to two tasks exceeded 90%, and average correctly added tasks per command was 79.28%.The authors report task-integration accuracy was influenced by the number of tasks in a command.Source: §VIII · p. 9
  • Task count significantly affected integration accuracy (F-statistic = 2.61, p-value = 0.035); fewer-task commands performed better, with a likely dataset-similarity uptick at six tasks.The authors attribute the slight improvement at six tasks as likely related to similarity with Eurobot strategies used to generate fine-tuning data.Source: §VIII · p. 9
  • 72.8% average accuracy, 4.71 relevance, and 4.89 informativeness; evaluator agreement was strong for accuracy but moderate for the other ratings.The authors attribute the agreement difference to the more subjective nature of relevance and informativeness.Source: §VIII · p. 9
  • The system combines dialogue, command-based BT construction for agents, and feedback on task execution in near-real-world trials.The conclusion describes a multi-agent system engaging with human commands and questions.Source: §VIII · p. 9
  • The authors identify commands with larger numbers of tasks as an area for further improvement, proposing decomposition into simpler tasks before BT generation.They suggest implementing decomposition in the BT-generation adapter or training a separate adapter for it.Source: §VIII · p. 9

IX. CONCLUSION

LLM-MARS uses a Falcon 7B-based model to manage multi-agent robots through dynamic dialogues. Eurobot 2023 trials report 79.28% average task execution accuracy, while expert evaluations found the system’s answers accurate, relevant, and informative.

IX. CONCLUSION

  • LLM-MARS uses a Falcon 7B-based LLM to manage multi-agent robotic systems through dynamic dialogues.The paper presents the system as a way for robots to generate behavior from operator commands and answer questions about their actions.Source: §IX · p. 9
  • 79.28% average task execution accuracy was achieved in trials under the Eurobot 2023 game rules.The conclusion also notes a correlation between task-integration accuracy and the number of tasks in a command.Source: §IX · p. 9
  • Expert evaluations found LLM-MARS’s answers to operators’ questions highly accurate, relevant, and informative.The conclusion identifies handling more complex commands as an area for future research, including command decomposition and specialized adapters.Source: §IX · p. 9
  • A single LLM gives a robot group AI capabilities and configures the system for specific tasks without specialized programming for each robot.The system can also answer inquiries about its performance.Source: §IX · pp. 9–10
  • The authors identify logistics, autonomous exploration, and human-machine collaboration in Industry 5.0 as potential application areas.The conclusion describes mobile robot groups as having potential to revolutionize logistics and robot or drone groups as candidates for autonomous exploration.Source: §IX · p. 10

Related papers

Loading 2312.09348v1…