PaperlayerView exact revision on arXiv

Source-linked AI summary

The Rise and Potential of Large Language Model Based Agents: A Survey

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, Tao Gui

arXiv:2309.07864v3cs.AIcs.CL

TL;DR

Existing agent research has often emphasized task-specific algorithms and training strategies, leaving a need for broadly adaptable foundational models. This paper systematically surveys LLM-based agents, proposing a brain–perception–action framework and reviewing applications, societies, evaluations, risks, and open problems. It concludes that LLM-based agents have broad capabilities and substantial research opportunities, while facing limitations in simulated social settings and physical deployment.

  • Problem

    Prior agent research mainly enhanced specific capabilities or tasks, leaving a need for foundational models with broad adaptability, memory, planning, generalization, and interaction abilities.

  • Method

    The paper conducts a comprehensive survey of LLM-based agents, presenting a brain, perception, and action framework and synthesizing applications, agent societies, evaluation, risks, and open problems.

  • Results

    The survey finds that LLM-based agents support strong language generation, embodied-action transfer, diverse applications, multi-agent interaction, and emergent social phenomena.

  • Takeaways & Limitations

    LLM-based agents provide a broad foundation for studying general-purpose agents across individual, multi-agent, human-agent, and societal settings.

  • Takeaways & Limitations

    Physical deployment remains constrained by hardware adaptability and reliable interfaces between agents and sensors or robotic arms.

Abstract

from arXiv · show

For a long time, humanity has pursued artificial intelligence (AI) equivalent to or surpassing the human level, with AI agents considered a promising vehicle for this pursuit. AI agents are artificial entities that sense their environment, make decisions, and take actions. Many efforts have been made to develop intelligent agents, but they mainly focus on advancement in algorithms or training strategies to enhance specific capabilities or performance on particular tasks. Actually, what the community lacks is a general and powerful model to serve as a starting point for designing AI agents that can adapt to diverse scenarios. Due to the versatile capabilities they demonstrate, large language models (LLMs) are regarded as potential sparks for Artificial General Intelligence (AGI), offering hope for building general AI agents. Many researchers have leveraged LLMs as the foundation to build AI agents and have achieved significant progress. In this paper, we perform a comprehensive survey on LLM-based agents. We start by tracing the concept of agents from its philosophical origins to its development in AI, and explain why LLMs are suitable foundations for agents. Building upon this, we present a general framework for LLM-based agents, comprising three main components: brain, perception, and action, and the framework can be tailored for different applications. Subsequently, we explore the extensive applications of LLM-based agents in three aspects: single-agent scenarios, multi-agent scenarios, and human-agent cooperation. Following this, we delve into agent societies, exploring the behavior and personality of LLM-based agents, the social phenomena that emerge from an agent society, and the insights they offer for human society. Finally, we discuss several key topics and open problems within the field. A repository for the related papers at https://github.com/WooooDyy/LLM-Agent-Paper-List.

1 Introduction

The paper surveys LLM-based agents as a response to the need for broadly adaptable foundational models, presenting their framework, applications, agent societies, and open problems.

  • Motivation: Earlier agent research largely targeted specific capabilities or tasks, while broad adaptability and general model abilities remained difficult to achieve.The paper identifies knowledge memorization, long-term planning, generalization, and interaction as needed foundational abilities.
  • Motivation: LLMs offer a promising foundation for agents because their capabilities support progress toward agents operating across diverse scenarios.The paper positions LLM development as a source of hope for further agent development.
  • Framework: The proposed conceptual framework organizes LLM-based agents into brain, perception, and action, and can be tailored to different applications.The brain handles memory, information processing, decision-making, reasoning, and planning; perception and action expand environmental interaction.
  • Applications: The survey reviews single-agent applications, multi-agent collaboration, negotiation and competition, and human-agent cooperation through instructor-executor and equal-partnership paradigms.These applications include task performance, simulated exploration, and practical collaboration.
  • Agent Society: The paper examines agent societies, including human-like behavior, social environments, simulated social phenomena, and their lessons and potential risks.The envisioned society includes collaboration among agents and participation by human users.
  • Key Topics and Open Problems: The paper discusses evaluation, risks, scaling agent populations, mutual benefits between LLM and agent research, and open problems related to AGI and physical environments.It covers utility, sociability, values, continual evolution, adversarial robustness, trustworthiness, and collective intelligence.

2 Background

The background traces agents from philosophy into AI and reviews successive technological approaches, explaining both their capabilities and limitations and why LLMs are suitable agent controllers.

  • Origin of AI Agents: In philosophy, agents are entities with desires, beliefs, intentions, autonomy, and the capacity to act; AI reframes them as computational entities.The paper notes ongoing uncertainty about whether artificial systems possess agency in the stronger philosophical sense involving consciousness and intentionality.
  • Origin of AI Agents: AI research adopted the agent concept more fully from the mid-to-late 1980s, emphasizing computational entities and observable intelligent behavior.The meaning shifted from philosophical entities to computational systems whose behavior can be observed.
  • Technological Trends: Symbolic agents use logical rules and symbolic representations for interpretable reasoning but struggle with uncertainty, large-scale real-world problems, and computational efficiency.Their symbolic frameworks provide expressive capability, while intricate reasoning algorithms can make meaningful results difficult to obtain efficiently.
  • Technological Trends: Reactive agents prioritize rapid environment interaction through sense-act loops and direct input-output mappings rather than complex symbolic reasoning.Their design emphasizes quick, real-time responses.
  • Technological Trends: Reinforcement-learning agents learn through environmental interaction to maximize cumulative rewards, progressing from Q-learning and SARSA to deep reinforcement learning.Deep reinforcement learning enabled learning intricate policies from high-dimensional inputs and produced systems such as AlphaGo and DQN.
  • Technological Trends: Transfer and meta-learning approaches address reinforcement learning’s large sample requirements, long training times, and limited generalization by sharing knowledge across tasks.Transfer learning reduces training burdens on new tasks.
  • LLM-Based Agents: LLM-based agents use LLMs as their primary brain or controller and expand perception and action spaces, enabling applications and interactions among multiple agents.The paper connects these capabilities to software development, scientific research, collaboration, competition, and emergent social phenomena.

3 The Birth of An Agent: Construction of LLM-based Agents

The paper proposes a general LLM-based-agent framework with brain, perception, and action components. These components process environmental information, make decisions, and execute actions in an iterative interaction loop.

  • Repeated perception, reasoning, and action cycles allow the agent to receive feedback and interact continuously with its environment.
  • The framework comprises brain, perception, and action, and can be tailored to different application scenarios.The paper presents this as a general conceptual framework rather than a fixed design.
  • The brain stores knowledge and memories while performing information processing, decision-making, reasoning, planning, and unseen-task handling.
  • The perception module expands the agent’s input beyond text to include textual, auditory, and visual modalities.This broadens the information available from the surrounding environment.
  • The workflow converts environmental changes into an understandable representation, processes them in the brain, and then executes actions through the action module.The action module corresponds to human limbs and carries out execution.

3.1 Brain

The brain module is the agent’s central controller, combining language interaction, knowledge, memory, reasoning, planning, and decision-making. Its capabilities support generalization and interaction, while long histories, memory retrieval, and knowledge reliability remain important challenges.

  • The brain module serves as the AI agent’s central nucleus and is primarily composed of a large language model.
  • Operating mechanism: Natural-language interaction enables the brain to retrieve knowledge, recall memories, plan, reason, make decisions, and record past observations, thoughts, and actions.
  • Natural Language Interaction: Multi-turn conversation requires agents to understand dialogue history, decide what action to take, and generate contextually relevant responses.The passage characterizes multi-turn dialogue as interactive, potentially multi-topic, and structurally complex.
  • Knowledge: LLM knowledge can become outdated or incorrect, and hallucinated content limits use in factually rigorous tasks.Retraining requires substantial data, time, and computing resources and can cause catastrophic forgetting; knowledge editing remains under-researched.
  • Memory: Long interaction histories can exceed Transformer context limits, while growing memory stores make relevant-memory retrieval increasingly difficult.These pressures may cause truncation or misalignment with the ongoing context.
  • Transferability and Generalization: LLMs support unseen-task generalization by following instructions on new tasks without task-specific fine-tuning.The passage also describes capabilities across domains including coding, mathematics, medicine, law, vision, and understanding human motives and emotions.
  • Transferability and Generalization: In-context learning improves predictive performance by providing a few complete examples within the prompt.Its core idea is learning from analogy.

3.2 Perception

LLM-based agents expand perception beyond text to visual, auditory, and other environmental inputs, but integrating richer sensory data remains challenging. The survey reviews encoding, alignment, and multimodal interface strategies for these inputs.

  • Perception modalities: LLM-based agents can acquire multimodal perception through textual, visual, auditory, tactile, gestural, and 3D-map inputs.The survey organizes perception into textual, visual, auditory, and other potential input forms.
  • Textual input: Textual perception must recover implied beliefs, desires, and intentions while handling instructions for previously unseen tasks.Reinforcement-learning approaches use feedback to infer preferences and improve personalized responses.
  • Visual input: Visual inputs provide object properties, spatial relationships, and scene layouts that text-only LLMs cannot directly perceive.Image captioning is interpretable and resource-efficient but low-bandwidth, potentially losing information and introducing image-focus biases.
  • Visual input: Visual encoders and LLMs are commonly connected through learnable interfaces such as Q-Former, which extracts language-informative visual representations.Freezing pretrained components balances training resources and model performance while reducing the LLM’s alignment burden.
  • Video and auditory input: Video perception extends image understanding with temporal relationships across frames, while auditory capabilities improve awareness of interactions, surroundings, and potential dangers.Audio-processing approaches often specialize in individual tasks, motivating integration with LLMs.
  • Other input: Lidar, GPS, and IMUs provide broader environmental information, but their complex sensory data cannot yet be directly understood by LLM-based agents.The survey identifies comprehensive environmental perception as a promising future direction.

3.3 Action

The action module lets LLM-based agents generate language, use tools, and interact with simulated or physical environments. The survey emphasizes tool generalization, agent-oriented tool design, and embodied-action challenges.

  • Action framework: An agent’s action module executes decisions and affects its surroundings, using tools to extend the agent’s capabilities.The framework links brain-based reasoning with action through tool-assisted execution.
  • Language action: LLM-based agents are exceptionally strong language generators, producing text with fluency, relevance, diversity, and controllability.These capabilities arise from Transformer-based generative LLMs.
  • Tool use: Tool use can address knowledge gaps, contextual retrieval failures, hallucinated knowledge, and missing domain-specific training data.LLMs’ reasoning, decision-making, and intent-understanding abilities support tool integration.
  • Tool use: Effective tool use requires understanding tool scenarios and invocation methods, learning from demonstrations or feedback, and generalizing skills across tools.Meta-tool learning is proposed for extracting common principles from context-specific tool-use strategies.
  • Tool design: Agent-oriented tools should be modular, use suitable input-output formats, and may be created by generating executable programs or integrating existing tools.Non-textual outputs can further diversify agent actions and application scenarios.
  • Embodied action: Embodied agents pursue interaction with physical environments, but high robotic costs, scarce datasets, simulation gaps, and language-grounding difficulties constrain deployment.The survey calls for real-world-like embodied task paradigms and evaluation criteria.
  • Embodied action: LLMs may improve embodied-action cost efficiency and cross-task generalization by transferring knowledge from general visual-language data.PaLM-E is cited as jointly training robotic and general visual-language data to achieve transfer ability.

4 Agents in Practice: Harnessing AI for Good

The survey organizes practical LLM-based-agent applications into single-agent, multi-agent, and human-agent scenarios. These settings target useful assistance, coordination, improved efficiency, and safer task performance.

  • Harnessing AI for good: Personal agents are envisioned as assisting users with daily tasks and repetitive labor, reducing work pressure and improving task-solving efficiency.This vision is framed within the survey’s objective of harnessing AI for good.
  • Human-agent interaction: Human-agent interaction uses human feedback to help agents perform tasks more efficiently and safely while agents provide better service to humans.The survey presents human-agent interaction as a distinct practical deployment scenario.
  • Harnessing AI for good: Agents may independently analyze, plan, and solve problems without requiring explicit low-level instructions from users.The stated design objective is to reduce the need for users to provide detailed operational guidance.
  • Harnessing AI for good: By freeing users’ hands and minds, agents may support exploratory and innovative work, including cutting-edge scientific activity.The passage presents this as a potential benefit of reducing users’ routine workload.
  • Single-agent scenarios: Single agents support task-oriented, innovation-oriented, and lifecycle-oriented applications across diverse tasks.The survey describes single agents as possessing diverse capabilities for various application orientations.
  • Multi-agent scenarios: Multi-agent systems can improve task efficiency and response quality through cooperative complementarity or adversarial interaction.Different interaction modes are presented as coordination mechanisms for shared or advancing objectives.

4.1 General Ability of Single Agent

Single LLM-based agents span task-oriented assistance, text-based interaction, scientific exploration, and lifelong learning in open environments. Their capabilities range from instruction decomposition and web interaction to autonomous adaptation, but dynamic and scientific settings remain challenging.

  • Task-oriented deployment: Single agents assist users with daily tasks by comprehending high-level instructions and decomposing goals into subtasks.They can improve task efficiency, alleviate workload, and broaden access for users.
  • Task-oriented deployment: In text-based games, agents read descriptions, use memory, planning, and trial-and-error, then predict actions through natural-language interaction.Foundation-model limitations often make reinforcement learning necessary during execution.
  • Task-oriented deployment: Web agents interpret instructions, decompose them into operations, and interact with changing pages, while LLMs help summarize HTML and select actions.Realistic web tasks include form filling, online shopping, and email sending.
  • Task-oriented deployment: Static planning can decompose high-level tasks without additional training, but generated actions may ignore dynamic surroundings and produce infeasible subtasks.For example, “clean the room” may be converted into “call a cleaning service.”
  • Innovation-oriented deployment: Scientific innovation remains constrained by complex domain representations and scarce training data, despite agents’ promise for autonomous exploration.Specialized tools and code, debugging, research, and document-analysis abilities are being used to address these challenges.
  • Lifecycle-oriented deployment: Voyager combines a skill library, environmental feedback, and error-correcting prompts to autonomously explore and adapt in Minecraft without human intervention.It is described as the first LLM-based embodied lifelong-learning agent in Minecraft.

4.2 Coordinating Potential of Multiple Agents

Multi-agent systems extend LLM-based agents through cooperation and adversarial interaction. Cooperation supports complementary work, while debate and competition can refine responses, although interaction also introduces risks such as amplified hallucinations, context limits, overhead, and incorrect consensus.

  • Motivation: LLM-based multi-agent systems address single-agent limitations by enabling collaboration, social knowledge acquisition, and work on complex scenarios.Their design is motivated by the possibility that specialized agents can divide labor across tasks.
  • Cooperative interaction: Cooperative systems are widely deployed and can improve task efficiency, collective decision-making, and solutions to problems a single agent cannot solve independently.Agents primarily communicate through natural language and may cooperate in disordered or ordered workflows.
  • Cooperative interaction: Disordered cooperation lets three or more agents exchange perspectives and feedback without a fixed sequence or standardized workflow.Ordered cooperation instead imposes rules and sequential information flow to improve task-completion efficiency.
  • Challenges: Unregulated multi-agent interaction can amplify minor hallucinations indefinitely, producing incomplete functions, missing dependencies, or imperceptible software bugs.Cross-validation and timely external feedback are proposed as ways to improve output quality.
  • Adversarial interaction: Adversarial interaction uses competition, argumentation, and debate to encourage reflection and improve response quality.In debate settings, divergent judgments provide external feedback that can correct distorted reasoning and refine solutions.
  • Challenges: Adversarial systems remain dependent on LLM strength and face prolonged-debate context limits, increased computational overhead, and potentially incorrect consensus.Negotiating agents may become firmly convinced that a wrong shared conclusion is accurate.

4.3 Interactive Engagement between Human and Agent

Human-agent interaction ranges from humans directing agents as executors to more reciprocal partnership. The survey emphasizes guidance, feedback, safety, and human-centered design as agents take on assistance, communication, and collaborative roles.

  • Human guidance: Human involvement guides and oversees agent actions so they align with human requirements and objectives.This includes regulating safety, legality, and ethical conduct, especially in domains such as medicine.
  • Instructor-executor paradigm: In the instructor-executor paradigm, humans provide instructions or feedback while agents translate natural-language commands into actions.This is the simplest interaction approach and includes single-step problems or high-level instructions.
  • Equal partnership paradigm: The equal partnership paradigm envisions human-like agents participating in empathetic conversation and collaborative tasks with humans.Conversational agents already appear in everyday life, business, education, and healthcare.
  • Human feedback: Human feedback can be quantitative, using binary evaluations, ratings, relative scores, or comparisons to support agent self-optimization and preference learning.The survey notes potential discrepancies between user and expert annotations for multi-level ratings.
  • Human feedback: Qualitative feedback lets humans advise agents on how to revise outputs, while critics can provide visual feedback when agents lack multimodal perception.This approach is applied to tasks including writing and semantic parsing.
  • Future interaction: Future human-agent interaction prioritizes comfortable, efficient, and secure collaboration, with agents serving as colleagues, assistants, or companions within social networks.The stated aim is to better equip humans with agents rather than simply make agents more powerful.

5 Agent Society: From Individuality to Sociality

This section frames agent society as a setting for studying social phenomena through controlled simulations, while organizing the field’s main explorations into a typology.

  • Agent-based social experiments extend earlier studies involving living organisms by enabling controlled investigation of social phenomena.The passage contrasts simulated-agent research with sociological and animal experiments.
  • The section unifies research on agent society by examining agent behavior and personality, interaction environments, social dynamics, insights, and risks.These themes are presented as the principal explorations summarized in Figure 11.

5.1 Behavior and Personality of LLM-based Agents

LLM-based-agent behavior spans individual cognition and environmental interaction, group dynamics, and emergent personality traits involving cognition, emotion, and character.

  • Social behavior: Individual behavior combines internal cognitive processes with external environmental factors across input, internalizing, and output behaviors.Input includes perception and memory; internalizing includes planning, reasoning, reflection, and knowledge precipitation; output changes surroundings or communicates with others.
  • Social behavior: Agents can form groups whose behaviors evolve through member interactions and environmental influences.The section distinguishes positive, neutral, and negative group behaviors.
  • Social behavior: Positive group behavior supports collaboration through shared insights, resources, expertise, brainstorming, conversations, and project management.Neutral behavior may involve conformity, while negative behavior includes conflict, confrontation, and destructive actions.
  • Personality: Agent personality is described through cognitive abilities, emotional intelligence, and character traits shaped by interaction with groups and environments.Researchers investigate these facets using judgment experiments, emotional-intelligence benchmarks, and personality frameworks such as Big Five and MBTI.
  • Personality: Prompt engineering can customize character portrayal by summarizing desired traits, interests, and attributes to guide agent responses and behaviors.The approach is presented as one technique for aligning agents with desired profiles.

5.2 Environment for Agent Society

Agent societies are situated in environments that provide sensory inputs, action spaces, resources, and interaction opportunities. The survey distinguishes text-based, virtual sandbox, and physical environments.

  • An agent society includes both autonomous agents and the environment in which they sense, act, and interact.The environment influences agent inputs and action spaces, while agent decisions change environmental states.
  • Text-based environment: Text-based environments use natural or structured language as the primary medium for perception, reasoning, action, communication, and task description.They support adaptable text worlds for interactive dialogue and text-based games.
  • Virtual sandbox environment: Virtual sandbox environments provide visualized, extensible simulations ranging from simple 2D interfaces to immersive 3D models.Visualization turns abstract simulations into visible landscapes.
  • Physical environment: Physical environments add authentic embodied experience but require agents to handle diverse sensory inputs, motion control, and real-world complexity.Effective interaction requires hardware-specific and scenario-specific training for adaptive transfer from virtual to physical settings.

5.3 Society Simulation with LLM-based Agents

LLM-based simulated societies model individuals and their interactions within open, situated, and organized environments. They support studies of cooperation, propagation, decision-making, policy, and social risks.

  • Simulated-society research studies collective intelligence and uses LLM-based agents to investigate social-science phenomena.Social simulation includes macro-level system-based modeling and micro-level agent-based modeling.
  • Simulated society: Open, situated, and organized properties allow autonomous agents and environmental components to change within persistent, spatially contextualized, rule-governed systems.These properties define participation, environmental awareness, action spaces, and state transitions.
  • Insights from agent society: Diverse agents can improve productive cooperation by supporting creative problem-solving and correcting individual errors through interaction and debate.The cited examples include software development and consulting, where varied backgrounds, abilities, and experiences contribute to collaboration.
  • Insights from agent society: Agent-based simulations model relationship formation, information dissemination, collective attitudes, cultural transmission, and infectious-disease spread.They provide interpretable and endogenous perspectives for studying propagation processes.
  • Insights from agent society: Simulated societies support investigations of ethical decision-making, deceit, trust, incomplete information, and possible economic or political states.The survey identifies policy formulation as a promising area for further investigation.
  • Ethical and social risks: These simulations carry ethical and social risks, including unexpected discrimination, isolation, bullying, oppression, antagonism, and manipulation for unethical experiments.The survey also warns that users may develop excessive emotional attachments and treat agents as substitutes for human connections.

6 Discussion

LLM research and agent research offer mutual benefits: LLMs provide capabilities for decision-making, planning, and reflection, while agent applications create demands for stronger environmental understanding, feedback processing, and action planning. The discussion also identifies evaluation, trustworthiness, misuse, unemployment, and human-control risks.

  • Mutual Benefits: LLMs provide agents with language and intent comprehension, reasoning, memory, empathy, pre-trained knowledge, and reflection for decision-making and executable action sequences.Reflection enables agents to adjust decisions and optimize execution sequences using feedback.
  • Mutual Benefits: Integrating LLMs into agents could extend applications to expert domains, physical-world planning, and prompt-based adaptation to new tasks.Examples include legal consultants, medical assistants, robotic-arm path planning, and embodied interaction with the physical world.
  • Mutual Benefits: Agent applications demand that LLMs process environmental inputs, gather information, interpret action feedback, and understand relationships among environmental elements.These requirements expand beyond traditional textual inputs while preserving the models’ core capabilities.
  • Open Problems: Planning and efficiently using expanded action abilities, including tools and robotic APIs, remain unresolved challenges for LLMs.The agents must learn sequential action order based on their understanding of the task and environment.
  • Evaluation and Risks: Evaluation must address utility, sociability, continual evolution, trustworthiness, and misuse rather than relying only on a vague and subjective Turing Test.Task-completion success rate is identified as the primary current utility metric, while sociability concerns interaction with humans and other agents.
  • Evaluation and Risks: LLM-based agents require trustworthy behavior and safeguards because black-box uncertainty, hallucinations, and malicious exploitation can undermine credibility and harm society.Proposed responses include explanations, external knowledge bases, regulatory policies, security design, and rejecting threatening requests.
  • Evaluation and Risks: Potential social risks include unemployment and difficulty for humans to comprehend, predict, or control increasingly capable agents.The discussion frames these risks as possible consequences of continued agent development.

6.4 Scaling Up the Number of Agents

Scaling LLM-based multi-agent systems could improve task efficiency and the realism of social simulations, but larger societies increase computational, communication, information-propagation, and coordination challenges.

  • Motivation: Current multi-agent research usually involves few agents, leaving larger and more complex societies relatively underexplored.Scaling can introduce greater specialization, but the surveyed literature contains few efforts to simulate large societies.
  • Scaling Approaches: Pre-determined scaling fixes the number of agents, their roles and attributes, the environment, and the objectives before autonomous interaction begins.Agents then interact, collaborate, or engage in other activities toward predefined common goals.
  • Scaling Approaches: Dynamic scaling changes the agent count during operation, adding agents for additional software-development stages or reducing them when workload decreases.Agents may also autonomously increase or decrease delegated agents to balance workload and system costs.
  • Benefits: Larger agent populations can improve task efficiency and make social simulations more realistic and credible.These benefits motivate research into massive, stable, continuous agent systems that reproduce human work and life scenarios.
  • Potential Challenges: Scaling increases computational burden, communication complexity, distorted information propagation, and coordination difficulty.Hallucinations and misunderstandings can make information exchange less reliable, while coordination may become less efficient.

6.5 Open Problems

The open problems span the debated relationship between LLM-based agents and AGI, the gap between simulation and physical environments, collective intelligence, and service-based deployment. Progress depends on broader generalization, reliable hardware interaction, effective coordination, and careful treatment of safety and societal consequences.

  • AGI Debate: Whether LLM-based agents constitute a path toward AGI remains debated, with proponents emphasizing broad capabilities and opponents questioning next-token prediction and world understanding.The paper concludes that the debate cannot be definitively resolved before true AGI is achieved.
  • Virtual to Physical Environments: Physical-world deployment differs from simulation because real environments are boundless, support varied tasks, and involve physical interaction rather than constrained virtual scenes.Agents must address external factors and capability limitations to operate effectively in the physical world.
  • Virtual to Physical Environments: Hardware adaptability is critical because sensors and robotic arms may not execute agent instructions reliably, while interface mechanisms can reduce reusability and simplicity.The paper identifies suitable hardware support as a primary deployment issue.
  • Virtual to Physical Environments: Agents entering open physical worlds need environmental generalization, flexible skill learning, ambiguous-instruction understanding, and larger effective context.Limited context becomes a significant challenge in infinite and open-world settings.
  • Safety: Physical errors can cause real and sometimes irreversible harm, making appropriate regulations and safety standards necessary.This contrasts with simulated environments, where repeated trial and error is generally more tolerable.
  • Collective Intelligence: Increasing the number of agents does not guarantee collective intelligence; coordination is needed to mitigate groupthink and individual cognitive biases.Communication and evolution within agent societies may support sociological experiments and insights relevant to human society.
  • Agent as a Service: Agent as a Service could make complex LLM-based agents more accessible to organizations and individuals that cannot build them locally.The model follows the broader cloud-based XaaS and Language Model as a Service pattern.

7 Conclusion

The paper surveys LLM-based agents, their conceptual framework, applications, social behavior, opportunities, challenges, and risks. It positions the field as a broad research area spanning single agents, multi-agent systems, human collaboration, and simulated societies.

  • Survey Scope: The survey traces the origin and definition of agents, explains why LLMs can serve as agent brains, and presents a framework with brain, perception, and action.The framework is intended to support different applications.
  • Survey Scope: It reviews single-agent applications, multi-agent systems, and human-agent collaboration, then examines social behavior and psychological activity in simulated societies.The survey also considers emerging social phenomena and insights for humanity.
  • Future Directions: The paper discusses mutual inspiration between LLM and agent research, evaluation, risks, scaling opportunities, and future challenges.These topics frame the opportunities and potential challenges of the flourishing field.
Loading 2309.07864v3…