PaperlayerView exact revision on arXiv

Source-linked AI summary

MindAgent: Emergent Gaming Interaction

Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, Jianfeng Gao

arXiv:2309.09971v2cs.AIcs.HCcs.MA

TL;DR

The paper addresses the shortage of benchmarks for general multi-agent collaboration, including LLM and human-NPC interaction. It introduces CUISINEWORLD and MINDAGENT for studying LLM planning and coordination, and reports potential to generalize coordination to more agents.

  • Problem

    The paper identifies a shortage of benchmarks for general multi-agent collaboration infrastructure that includes both LLM and human-NPC collaboration.

    Source: §1 · p. 2

  • Method

    The paper introduces CUISINEWORLD and MINDAGENT, using in-context demonstrations, action rationales, and environmental feedback to study LLM multi-agent planning.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    Experimental results support LLMs’ potential as generalist multi-agent planners, including generalization to coordinating more agents.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    MINDAGENT is deployed across gaming domains to demonstrate multi-agent and human-AI collaboration in real-world video games.

    Source: §8 · p. 15

    The authors note that LLM multi-agent planning can be constrained by computation cost, context length, and non-optimal plans.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large Language Models (LLMs) have the capacity of performing complex scheduling in a multi-agent system and can coordinate these agents into completing sophisticated tasks that require extensive collaboration. However, despite the introduction of numerous gaming frameworks, the community has insufficient benchmarks towards building general multi-agents collaboration infrastructure that encompass both LLM and human-NPCs collaborations. In this work, we propose a novel infrastructure - MindAgent - to evaluate planning and coordination emergent capabilities for gaming interaction. In particular, our infrastructure leverages existing gaming framework, to i) require understanding of the coordinator for a multi-agent system, ii) collaborate with human players via un-finetuned proper instructions, and iii) establish an in-context learning on few-shot prompt with feedback. Furthermore, we introduce CUISINEWORLD, a new gaming scenario and related benchmark that dispatch a multi-agent collaboration efficiency and supervise multiple agents playing the game simultaneously. We conduct comprehensive evaluations with new auto-metric CoS for calculating the collaboration efficiency. Finally, our infrastructure can be deployed into real-world gaming scenarios in a customized VR version of CUISINEWORLD and adapted in existing broader Minecraft gaming domain. We hope our findings on LLMs and the new infrastructure for general-purpose scheduling and coordination can help shed light on how such skills can be obtained by learning from large language corpora.

1 INTRODUCTION

The paper addresses the complexity of coordinating multiple agents toward shared goals and introduces CUISINEWORLD and MINDAGENT to study LLM multi-agent planning. Its evaluations report generalization to more agents, benefits from prompting techniques, and human-AI gaming interactions.

1 INTRODUCTION

  • Without elaborate prompting, pretrained LLMs such as GPT-4 can schedule 2–4 agents to complete dishes and collaborate with human players using game instructions and recipes.The introduction describes these as zero-shot multi-agent planning observations.Source: §1 · p. 2
  • CUISINEWORLD is a virtual-kitchen benchmark with a minimal text-based format and varied planning task structures and difficulty levels.The benchmark is intended to test LLM scheduling and coordination capacity.Source: §1 · p. 2
  • MINDAGENT uses few-shot demonstrations, action rationales, and environmental feedback to facilitate LLM multi-agent planning.The framework follows in-context learning and evaluates multiple LLMs, including GPT-4, Claude, and LLaMA.Source: §1 · p. 2
  • Experimental results support LLMs’ potential as generalist multi-agent planners, particularly in generalizing coordination to more agents.The introduction identifies generalization to more agents as an evaluation finding.Source: §1 · p. 3
  • The system is deployed in real-world gaming scenarios to demonstrate human-AI interactions.The contribution statement describes deployment in real-world gaming scenarios.Source: §1 · p. 3

2 RELATED WORK

Prior work spans multi-agent collaboration, LLM task planning, and game benchmarks, but the paper distinguishes its focus on efficient coordination across multiple tasks and agents. It also contrasts its game setting and centralized system with prior approaches.

2 RELATED WORK

  • Traditional multi-agent collaboration has commonly been modeled with MDP/POMDP frameworks.The passage cites Lowe et al. (2017), Rashid et al. (2020), and Jain et al. (2019).Source: §2 · p. 3
  • Earlier LLM collaboration studies often focus on task success for a singular objective, whereas this work emphasizes collaboration efficiency across multiple objectives and more than two agents.The paper also contrasts its centralized system with individually simulated agents, noting fewer API calls and reduced context length.Source: §2 · p. 3
  • Prior LLM planning work decomposes natural-language instructions into subtasks, sometimes using low-level controllers and environmental feedback.The cited examples include robotics planning approaches.Source: §2 · p. 3
  • Unlike game settings with dominant and supporting roles or only two agents, this work assumes equal agent responsibilities and includes human-player collaboration.The paper also targets multiple distinct tasks with competing resources within one episode and varied task difficulty.Source: §2 · p. 3
  • The paper contrasts its game with Carroll et al. (2019), citing diverse tools and objectives that generate an exponentially larger task space.The comparison with related games is shown in Table 1.Source: §2 · p. 3

3 THE NEW GAMING CUISINEWORLD DESIGN AND BENCHMARK

CUISINEWORLD is a modular virtual-kitchen benchmark for evaluating multi-agent scheduling and coordination through dish orders, tools, and varied task difficulty. It formalizes dispatching and order completion in an interactive game and measures collaboration across order-arrival conditions.

3 THE NEW GAMING CUISINEWORLD DESIGN AND BENCHMARK

  • CUISINEWORLD is a modular virtual-kitchen game in which agents coordinate to complete as many dish orders as possible.Its task and game-engine design allows additional dish types and implementations such as text, Unity, or Minecraft.Source: §3 · p. 3

3.1 TASK DEFINITION

  • The game is formulated as a centralized, fully observable decision problem whose state aggregates descriptions of locations and agents.Location descriptions indicate contents or occupancy, while agent descriptions indicate location, held items, or tool use.Source: §3.1 · pp. 4–5
  • Each dispatch supplies one command per agent, executed sequentially, while orders arrive periodically and expire if unfinished.Commands include goto, get, put, activate, and noop; feedback and rewards or penalties are provided for events such as completion or infeasible commands.Source: §3.1 · p. 5

3.2 IMPLEMENTING CUISINEWORLD

  • The basic game has 10 location types, 27 ingredient types, 33 dishes, and 12 levels grouped into four difficulty classes.The levels are entry, simple, intermediate, and advanced, with three levels in each class; recipes, dishes, and levels can be extended.Source: §3.2 · p. 5

3.3 EVALUATION METRIC

  • CoS averages task-completion rates across τint conditions, with M = 5 in the default setting.The selected intervals span a wide difficulty range, from relaxed to intense scenarios.Source: §3.3 · pp. 5–6

4 MINDAGENT: INFRASTRUCTURE FOR GAMING AI

MINDAGENT combines in-context prompting and environmental information with a task-allocation objective for coordinating multiple agents. Its mechanism frames task assignment as utility maximization under time and feasibility constraints.

4.1 INFRASTRUCTURE

  • MINDAGENT is an in-context-learning framework built around a prompt, the current environment state, and interaction memory.Its prompt includes recipes, general instructions, inference knowledge, and a one-shot demonstration.Source: §4.1 · p. 6
  • The prompt supplies dish procedures, game rules, helpful hints, and a stepwise example of preparing a different dish.The demonstration covers ingredient collection, cooking, transporting intermediate products, and serving the finished dish.Source: §4.1 · pp. 6–7
  • The current state reports agent locations, possessions, available tools, and tool contents; memory records past environment and agent states.Natural-language feedback and a fixed-horizon history help address invalid actions and context-length limits.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 INFRASTRUCTURE MECHANISM

  • The mechanism models unknown tasks as expiring sequences of subtasks and seeks to complete more tasks while failing fewer within the available time.Subtask utility combines allocation quality and cost, with assignment defined for an agent, subtask, and task.Source: §4.2 · p. 7
  • Task decomposition uses binary assignments, limits each subtask to one agent, and maximizes episode utility under a time constraint.The paper notes that the formulated problem cannot be solved in polynomial time and uses LLMs, with natural-language feedback and selected history, to address it.Source: §4.2 · pp. 7–8

5 EXPERIMENTS AND RESULTS

The experiments examine multi-agent dispatch, prompting and model settings, and human collaboration in CUISINEWORLD. More agents generally improve dispatcher-side efficiency, but human evaluations also reveal an enjoyment trade-off as team size increases.

5.1 LLM SETTINGS

  • Experiments used GPT-4, ChatGPT, and Llama 2 settings at temperature 0.1, reporting averages over three episodes.GPT-4 used gpt-4-0613, ChatGPT used gpt-3.5-turbo-0613, and Llama 2 used Llama-2-70b-chat-hf.Source: §5.1 · p. 8

5.2 EXPERIMENT SETTING I: LLMS DISPATCH MULTI-AGENTS (NPC)

  • Across task levels, collaboration efficiency generally rose with more agents, but performance degraded with more agents under less demanding conditions with fewer tasks.CoS was highest with two agents in two cases, three agents in seven cases, and four agents in three cases.Source: §5.2 · pp. 8–9

5.3.1 HUMAN DATA COLLECTION

  • The human study recorded task successes and failures, preserved action histories for replay, and collected post-episode engagement ratings.Its objective measure assessed human-AI teaming performance, while the survey measured users’ perceptions.Source: §5.3.1 · p. 9

5.3.2 EXPERIMENT II SETTING

  • The user study compared participants working alone with collaboration alongside one, two, or three agents on an unseen level-three task.The study targeted the dispatcher’s ability to collaborate with human players.Source: §5.3.2 · p. 10

5.3.3 EXPERIMENT II DESIGN

  • LLM-agent teams had higher success than solo players (p < 0.001); collaboration also increased reported fun (p = 0.0126) and productivity (p = 0.0104).The 12-person study used a within-subject design with randomized treatment order and ANOVA followed by Tukey HSD comparisons.Source: §5.3.3 · pp. 10–11
  • More agents improved task success but reduced enjoyment, with players often feeling sidelined; participants also rated intelligent agents more predictable than random agents (p < 0.001).Human players’ actions were influenced by other players’ actions (p = 0.00266).Source: §5.3.3 · p. 11

5.4 VISUALING ”CUISINEWORLD”

  • CUISINEWORLD was implemented as a game with keyboard or VR interaction, including human-NPC collaboration and human players instructing agents.The figure depicts agents preparing mixed juice, a human player acting as head chef, and VR collaboration.Source: §5.4 · p. 12, Figure 6 · p. 12

6 ANALYSIS AND EMERGENT GAMING ABILITIES

The analysis finds that feedback and human-crafted inference knowledge support dispatcher performance, while short demonstrations can still support unseen tasks. GPT-4 also shows emergent task adaptation and can coordinate more agents than it saw in demonstrations.

6.1 ABLATION STUDY FOR MULTI-AGENTS

  • Excluding environmental feedback significantly reduces performance, while human-crafted inference knowledge improves it and a two-step demonstration can still support unseen tasks.Without feedback, the dispatcher tends to repeat mistakes and remain stuck in particular states; the two-step demonstrations omit explicit task-completion guidance.Source: §6.1 · p. 13
  • Other tested LLMs tend to underperform, although Claude-2 still completes the task to a considerable extent.The comparison used identical prompt inputs across models.Source: §6.1 · p. 13

6.2 EMERGING CAPABILITIES

  • With only a partial two-step demonstration, GPT-4 can handle unseen tasks and dynamically prioritize multiple tasks as they arise.The paper describes this as emergent zero-shot multi-agent planning and on-the-fly planning.Source: §6.2 · p. 13
  • GPT-4 can dispatch four agents after demonstrations involving two agents, and collaboration efficiency is higher with more agents.The paper presents this as emergent multi-agent reasoning.Source: §6.2 · p. 13

7 NOVEL GAME ADAPTATION

The authors adapt MINDAGENT to Minecraft, where agents perform cooking tasks, and report that the infrastructure can extend across distinct games. They also add voice interaction between human players and in-game agents, with a VR integration.

7 NOVEL GAME ADAPTATION

  • In Minecraft, Alex and Steve gather and cook meat, then deposit it in a chest using defined multi-agent actions and game-state information.For reproducibility, killed mobs and mined blocks respawn nearby.Source: §7 · p. 14, Figure 7 · p. 15
  • Game-session data provides evidence that the collaboration infrastructure can be adapted across multiple distinct games.The authors frame this as robustness supporting broader gaming applications.Source: §7 · p. 14
  • Azure speech-to-text enables human players to communicate goals to Minecraft NPC agents through real-time voice chat, and the infrastructure is integrated with Minecraft VR.The prompt includes additional human instructions and dialogue history when a human player chats with the multi-agent system.Source: §7 · p. 14

8 CONCLUSION

The conclusion presents MINDAGENT as an infrastructure for LLM-based multi-agent collaboration across gaming domains, with deployments in video games. The appendix provides examples of its system prompt and partial demonstration.

8 CONCLUSION

  • MINDAGENT is presented as an infrastructure for multi-agent collaboration through LLMs across multiple gaming domains.The authors report investigating its planning capabilities and deploying it in video games for multi-agent and human-AI collaboration.Source: §8 · p. 15
  • The conclusion frames seamless and intuitive human-AI collaboration as a direction for future gaming systems.The authors also hope their findings inform games that are more engaging and enjoyable.Source: §8 · p. 15

A PROMPT EXAMPLES

B TASK DETAILS IN CUISINEWORLD

CUISINEWORLD task graphs vary in complexity, and the environment can be extended by editing JSON files.

B TASK DETAILS IN CUISINEWORLD

  • Users can add or modify CUISINEWORLD tasks by changing JSON files, while task graphs represent different levels of complexity.The environment is described as customizable and extendable.Source: B TASK DETAILS IN CUISINEWORLD · p. 21

B.1 LEVEL 0

The examples name food tasks across several levels and identify Minecraft cooking tasks alongside their task graphs.

B.1 LEVEL 0

B.6 LEVEL 5

  • Level 5 is represented by a Tomato Cucumber Salad example.Source: §B.6 · p. 23

B.8 LEVEL 7

  • Level 7 is represented by an onionPotatoCarrotSoup example.Source: §B.8 · p. 24

B.9 LEVEL 8

  • Level 8 is represented by an onionBroccoliCheeseSoup example.Source: §B.9 · p. 24

C MINECRAFT

  • Minecraft coverage labels four cooking tasks—chicken, mutton, steak, and porkchop—and introduces their task graphs.Source: C MINECRAFT · p. 27

D HUMAN EVALUATION INTERFACE

The human evaluation interface is designed to isolate users’ perception of collaborative agents from control skill and GPT-4 latency.

Related papers

Loading 2309.09971v2…