Source-linked AI summary
ProAgent: Building Proactive Cooperative Agents with Large Language Models
Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, Xiaojun Chang, Junge Zhang, Feng Yin, Yitao Liang, Yaodong Yang
TL;DR
Learning-based cooperative agents can struggle to generalize to unfamiliar teammates, motivating ProAgent, an LLM-based framework for adaptive coordination. ProAgent reasons over language-based states, infers and corrects beliefs about teammate intentions, and validates and executes cooperative plans. Experiments in Overcooked-AI report superiority over alternative agents across diverse teammate settings, with interpretable reasoning and planning.
Problem
Learning-based cooperative agents depend heavily on training-time teammate diversity, limiting strategic adaptation when coordinating with unfamiliar teammates.
Method
ProAgent uses LLM-based language reasoning to analyze states, infer teammate intentions, update beliefs from subsequent behavior, and adapt cooperative actions.
Results
ProAgent outperforms other agents when cooperating with diverse teammate types in Overcooked-AI experiments.
Takeaways & Limitations
ProAgent provides an interpretable framework for cooperative coordination based on natural-language reasoning and planning.
Abstract
from arXiv · showhide
Building agents with adaptive behavior in cooperative tasks stands as a paramount goal in the realm of multi-agent systems. Current approaches to developing cooperative agents rely primarily on learning-based methods, whose policy generalization depends heavily on the diversity of teammates they interact with during the training phase. Such reliance, however, constrains the agents' capacity for strategic adaptation when cooperating with unfamiliar teammates, which becomes a significant challenge in zero-shot coordination scenarios. To address this challenge, we propose ProAgent, a novel framework that harnesses large language models (LLMs) to create proactive agents capable of dynamically adapting their behavior to enhance cooperation with teammates. ProAgent can analyze the present state, and infer the intentions of teammates from observations. It then updates its beliefs in alignment with the teammates' subsequent actual behaviors. Moreover, ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various of coordination scenarios. Experimental evaluations conducted within the Overcooked-AI environment unveil the remarkable performance superiority of ProAgent, outperforming five methods based on self-play and population-based training when cooperating with AI agents. Furthermore, in partnered with human proxy models, its performance exhibits an average improvement exceeding 10% compared to the current state-of-the-art method. For more information about our project, please visit~\url{https://pku-proagent.github.io}.
Introduction
LLM-based agents are extended from individual decision-making to cooperative multi-agent coordination, where unfamiliar teammates expose limits in learning-based policy generalization. ProAgent addresses this challenge by combining language-based reasoning, intention inference, and adaptive cooperation.
- LLM-based agents have mainly been studied as individual agents, leaving their potential for multi-agent coordination comparatively unexplored.
- Learning-based cooperative agents depend heavily on teammate diversity during training, limiting strategic adaptation to unfamiliar teammates in zero-shot coordination.
- ProAgent integrates LLM reasoning and planning into cooperative multi-agent settings as a framework for adaptive coordination.
- ProAgent interprets scenes, infers teammate intentions, and dynamically adapts behavior through an interpretable cooperative process.
- ProAgent outperforms other agents when cooperating with diverse teammate types, according to the paper’s experiments.
Related Works
Prior work on LLM reasoning and multi-agent coordination provides foundations for ProAgent, but existing cooperative-game approaches differ in planning centralization and teammate information use.
- Reasoning and Planning with Large Language Models: LLM reasoning methods decompose complex queries into sequential intermediate steps or simpler subproblems before producing solutions.
- Reasoning and Planning with Large Language Models: LLM-based cooperative-game research remains limited, with prior methods using either centralized planning or decentralized planning with explicit communication.
- Multi-agent Coordination: Multi-agent coordination seeks effective collaboration toward shared goals, but traditional approaches struggle with fixed task settings, multiple tasks, and unseen scenarios.
Method
ProAgent uses a modular, language-centered pipeline that converts task states into descriptions, reasons about teammate intentions, validates high-level skills, executes low-level actions, and updates memory from interaction.
- ProAgent continuously interacts with the environment through a hierarchical inference pipeline organized into multiple stages.
- The Knowledge Library and state grounding convert task rules, layouts, and raw tensor states into language descriptions that the LLM can process.
- Belief Correction updates beliefs about teammate intentions using subsequent behavior, improving the accuracy of future predictions.
- The Verificator checks whether a selected skill is feasible and supports replanning after failures through multi-round analysis.
- The task prompt specifies objectives, legal skills, and response structure, while optional demonstrations provide examples of desired analysis and behavior.
- The Memory module stores persistent task knowledge and recent trajectories to support informed cooperative decisions.
- The Planner applies Chain-of-Thought reasoning to analyze the current state, predict teammate intentions, and formulate a high-level skill.
Experiments
ProAgent is evaluated in Overcooked-AI with AI teammates, human proxies, and ablations of its reasoning modules. It generally performs strongly through adaptive planning, action adjustment, and feedback-based correction.
- AI teammates: ProAgent outperforms the baselines in all layouts when acting as Player 0, with only limited losses as Player 1.It slightly underperforms FCP in Cramped Room and loses to PBT in Forced Coordination.
- AI teammates: ProAgent requires no specific training with distinct teammates or layouts, indicating stronger adaptive ability than the other AI agents.The comparison includes SP, PBT, FCP, MEP, and COLE.
- Qualitative behaviors: ProAgent makes strategic plans, recognizes failed plans, and often produces better replacements.One example prioritizes supplying an onion to a second pot before fetching a plate while the first pot finishes cooking.
- Qualitative behaviors: ProAgent dynamically adjusts low-level actions while executing high-level plans, including rerouting around teammate-caused blockages.When no clear goal exists, the Controller can move randomly to help break deadlocks caused by other agents’ learned conventions.
- Ablations: 204 with both analysis and belief, 184 without belief, and 100 with neither analysis nor belief show progressively lower Cramped Room scores.The authors attribute the gains to analysis providing in-context information for planning and intention inference providing further improvements.
- Ablations: 20% success without the Verificator Module shows the importance of feedback when the Planner produces inaccurate plans.The success rate was computed over 100 steps.
Conclusion
The paper proposes ProAgent, an LLM-based framework for multi-agent coordination that combines reasoning and planning with adaptive behavior. Overcooked-AI experiments with AI agents and human proxies demonstrate effectiveness over state-of-the-art methods, while its natural-language reasoning remains interpretable.
- ProAgent is an LLM-based framework designed to address multi-agent coordination.
- The framework uses LLM common-sense comprehension, language-based task understanding, and explicit reasoning and planning.
- Experiments with AI agents and human proxies in Overcooked-AI demonstrate ProAgent’s effectiveness over state-of-the-art methods.
- ProAgent’s natural-language reasoning and planning are interpretable and friendly to humans.