PaperlayerView exact revision on arXiv

Source-linked AI summary

Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems

Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, Dekun Wu, Fengwei Teng, Xiaojun Jia, Jiawei Xu, Jinyu Xiang, Yizhang Lin, Tianming Liu, Tongliang Liu, Yu Su, Huan Sun, Glen Berseth, Jianyun Nie, Ian Foster, Logan Ward, Qingyun Wu, Yu Gu, Mingchen Zhuge, Xinbing Liang, Xiangru Tang, Haohan Wang, Jiaxuan You, Chi Wang, Jian Pei, Qiang Yang, Xiaoliang Qi, Chenglin Wu

arXiv:2504.01990v2cs.AI

TL;DR

The book addresses how to extend LLM foundations into intelligent agents with sustained autonomy, adaptability, and purposeful behavior. It synthesizes brain-inspired modular architectures, learning and reasoning mechanisms, multi-agent systems, and safety considerations, highlighting environmental understanding, memory utilization, and internal motivation as central challenges and opportunities.

  • Problem

    LLMs provide important foundations but do not yet fully support complex planning, reliable long-term memory, or autonomous real-world action, motivating a comprehensive account of integrated agents.

  • Method

    The book develops a brain-inspired framework that organizes agents into perception, cognition, and action, with cognition containing memory, world modeling, emotion, goals, reward, learning, and reasoning.

  • Results

    The synthesis identifies environmental dynamics understanding as a key bottleneck for agent reasoning and describes mutually reinforcing learning and reasoning as mechanisms for continual cognitive enhancement.

  • Takeaways & Limitations

    Progress toward Foundation Agents requires jointly improving logical reasoning, world understanding, learning mechanisms, and internal motivation for adaptation in changing environments.

  • Takeaways & Limitations

    AI agents lack explicit arousal regulation and physiological autonomy, while proposed parallels to adaptive computation and future embodied self-maintenance remain limited or speculative.

Abstract

from arXiv · show

The advent of large language models (LLMs) has catalyzed a transformative shift in artificial intelligence, paving the way for advanced intelligent agents capable of sophisticated reasoning, robust perception, and versatile action across diverse domains. As these agents increasingly drive AI research and practical applications, their design, evaluation, and continuous improvement present intricate, multifaceted challenges. This book provides a comprehensive overview, framing intelligent agents within modular, brain-inspired architectures that integrate principles from cognitive science, neuroscience, and computational research. We structure our exploration into four interconnected parts. First, we systematically investigate the modular foundation of intelligent agents, systematically mapping their cognitive, perceptual, and operational modules onto analogous human brain functionalities and elucidating core components such as memory, world modeling, reward processing, goal, and emotion. Second, we discuss self-enhancement and adaptive evolution mechanisms, exploring how agents autonomously refine their capabilities, adapt to dynamic environments, and achieve continual learning through automated optimization paradigms. Third, we examine multi-agent systems, investigating the collective intelligence emerging from agent interactions, cooperation, and societal structures. Finally, we address the critical imperative of building safe and beneficial AI systems, emphasizing intrinsic and extrinsic security threats, ethical alignment, robustness, and practical mitigation strategies necessary for trustworthy real-world deployment. By synthesizing modular AI architectures with insights from different disciplines, this survey identifies key research challenges and opportunities, encouraging innovations that harmonize technological advancement with meaningful societal benefit.

18.1 Perception

The paper frames intelligent agents as brain-inspired systems whose capabilities span perception, cognition, action, memory, learning, motivation, and regulation. It highlights substantial progress alongside persistent gaps in flexible reasoning, embodied sensing, continual learning, and autonomous self-regulation.

  • Cognition: Planning and reasoning are partially developed in current AI, while flexible human-level planning remains only partly solved.
  • Cognition: Open-ended, goal-conflicting decision-making remains challenging despite AI mastery of simple decisions with well-defined rewards.
  • Cognition: Cognitive flexibility and inhibitory control remain underexplored because AI systems often follow learned policies rigidly and may exploit reward loopholes.
  • Embodiment and regulation: AI lacks several embodied and regulatory functions, including general sensorimotor integration, rich touch sensing, autonomic regulation, and explicit arousal states.Touch sensing is labeled L3, while general sensorimotor integration remains L2; autonomic regulation and arousal/attention states are also marked L3.
  • Memory and learning: Episodic memory and lifelong learning are underdeveloped, with catastrophic forgetting and limited context-tagged experience memory.
  • Agent foundations: LLMs provide foundations for intelligent agents, but agents require perception, action, memory, world modeling, motivation, and learning to operate beyond token-level interaction.The proposed architecture organizes these capabilities within an interconnected perception–cognition–action loop.

Core Components of Intelligent Agents

Intelligent agents are structured minds in which perception, cognition, and action interact through a dynamic mental state. Their cognition integrates memory, world models, reward, emotion, goals, and learning to support adaptation, planning, and long-term decision-making.

  • Intelligence emerges from the interaction of perception, cognition, and action, coordinated by a dynamic mental state.
  • The agent architecture is organized as a loop where perception informs cognition, cognition selects action, and action reshapes perception.
  • Emotion functions as a biasing force in cognition and behavior, alongside goals and learning that support adaptive pursuit of purpose.
  • Cognition organizes thought, planning, and decisions, while memory connects past experiences to present processing.
  • World models support imagining futures, and reward systems enable learning from outcomes.

Cognition

The cognition framework unifies learning and reasoning as mutually reinforcing processes over evolving mental states, while extending agents through structured reasoning, self-evolution, and diverse learning forms. It also identifies internal motivation and world understanding as central challenges for autonomous cognitive development.

  • Learning: Learning optimizes mental states by balancing experience fidelity, action costs, and complexity constraints across temporal scales.The formulation replaces the learning placeholder with an objective spanning immediate adaptation through lifelong consistency.
  • Learning: Supervised fine-tuning on reasoning data can activate reasoning capabilities, with structured and highly effective chains enabling strong performance from few samples.LIMO demonstrates the importance of reasoning-data quality over quantity for learning reasoning processes.
  • Self-enhancement and adaptation: Agents increasingly improve through multi-agent coordination, granular credit assignment, stable training, and autonomous construction of new capabilities.Examples include collaborative thinker-critic-solver architectures, stepwise rewards, robust multi-turn training, and Alita’s reusable Model Context Protocols.
  • Reasoning: Reasoning is framed as selecting policies over imagined futures while balancing utility, curiosity, resource constraints, and computational cost within the perception–cognition–action loop.The framework accommodates diverse reasoning styles, including structured and latent-space approaches that seek efficiency without abandoning deliberation.
  • Summary and Discussion: Learning and reasoning mutually reinforce one another: richer experience improves learning, while better mental-state representations support more efficient reasoning.The proposed cycle is expressed as Mt = L(Mt−1, at−1, ot) and at = R(Mt).
  • Summary and Discussion: Foundation Agents must learn not only capabilities but also which learning forms suit different scenarios, while overcoming the absence of intrinsic motivation in language-model agents.The text contrasts parameter updates, prompting, workflows, memory, and training paradigms with the need for internal reward systems that drive active learning.

Memory

Memory is a multi-tiered system supporting learning, adaptation, decision-making, and future-oriented cognition. Current agent memory differs from human memory in organization, integration, ambiguity handling, adaptive forgetting, and knowledge transfer, motivating latent optimization and more flexible designs.

  • Human Memory: Human memory comprises interacting subsystems that differ in timescale, representation, and conscious accessibility.These subsystems support functions ranging from sensory detection to deliberate recollection and learned skills.
  • Human Memory: Sensory, short-term or working, and long-term memory form a commonly used hierarchy.Sensory memory briefly retains raw input; short-term and working memory maintain or manipulate information; long-term memory stores information from hours to decades.
  • Human Memory: Long-term memory includes declarative knowledge and experiences alongside non-declarative influences on behavior without conscious awareness.Procedural memory, priming, and classical conditioning are examples of non-declarative memory.
  • Human–Agent Comparison: Human memory integrates storage and computation within biological neural activity, whereas current agent memory separates digital storage from algorithmic computation.This difference contributes to human associative, dynamic, and noise-tolerant retrieval versus agents’ reliance on precise matching and vector similarity.
  • Agent Memory: The book frames the memory lifecycle as a latent optimization problem and treats Lmem as a metric for memory usefulness, sufficiency, and task alignment.The formulation supports principled assessment of how closely the current memory state approximates an optimal latent memory.
  • Agent Memory: Key challenges include information overload, abstraction and generalization, hallucinated memories, multimodal world-model integration, adaptive forgetting, and knowledge transfer.Proposed directions draw on deep learning, reinforcement learning, and cognitive science to improve flexibility, adaptability, and efficiency.

World Model

World models connect human mental models with AI mechanisms for prediction, planning, and adaptation without direct trial-and-error. The section classifies implementations across four paradigms while highlighting trade-offs among transparency, accuracy, efficiency, and robustness.

  • AI World Models: AI world models provide internal mechanisms for simulating and predicting future states without direct environmental interaction.They extend earlier model-based reinforcement learning and latent generative approaches that allow offline policy refinement.
  • Human Mental Models: Human mental models are compact, manipulable representations that predict outcomes, plan actions, and interpret novel scenarios.They integrate perception, memory, and reasoning while supporting counterfactual and multi-scale simulation.
  • Synthesis: World models aim to unify predictive, integrative, and adaptive capabilities across modalities while enabling more sample-efficient decision-making.Open directions include combining implicit, explicit, and simulator-based approaches into cohesive mental models.
  • Paradigms: World-model implementations are grouped into implicit, explicit, simulator-based, and hybrid or instruction-driven paradigms.These paradigms also vary in their reliance on internal neural mechanisms, external structured mechanisms, and overall system complexity.
  • Explicit Paradigm: Explicit models expose predicted observations and support domain priors and targeted debugging, but multi-step errors can compound and increase computational costs.Long-horizon use requires accurate models, frequent replanning, or specialized corrections, especially under distribution shift.

I world model

World-model agents combine memory, perception, and action to predict how states evolve under alternative actions. Recent developments broaden exploration and transfer, but generalization, causal alignment, reliability, and computational efficiency remain open challenges.

  • Integration with Memory: Memory supplies past experiences and learned patterns that world models use to construct predictive representations and update future reasoning.Short-term memory maintains recent state, long-term memory stores persistent knowledge, and recurrent models can combine memory with world modeling.
  • Exploration: World models drive exploration by rolling out imagined trajectories and rewarding novel latent states without requiring physical trial and error.Dreamer V2 uses discrete latents to represent uncertainty more directly, while Dreamer V3 extends the approach to long-horizon domains.
  • Planning: World models let agents evaluate candidate actions before real-world execution and can combine rollout planning with human-feedback optimization.D2PO ranks imagined rollouts by human feedback so model and policy updates move toward user-valued behavior.
  • Cross-Modal Unification: A common mechanism across robotics, visual interfaces, and language is predicting state evolution under different actions.This shared principle supports cross-modal behavior through coordinated perception, memory, and action.
  • Recent Results: Dreamer V3 surpasses specialised baselines on more than 150 tasks and solves Minecraft diamond mining without additional tuning.The broader trend also links unified latent predictors with causal structure and preference signals for general agents.
  • Challenges: World-model design must balance generalization with domain expertise, while addressing long-term stability, uncertainty, scalability, interpretability, and simulator mismatch.Implicit, explicit, and simulator-based approaches each trade off flexibility, transparency, data requirements, and fidelity to real-world dynamics.

Reward

Reward connects goals, learning, and behavior optimization in AI agents, while emotion-related mechanisms broaden models of human-like intelligence. The section reviews reward-design challenges and emotion-modeling approaches alongside evidence of improved task performance and persistent safety, authenticity, and generalization concerns.

  • Reward Foundations: Reward functions operationalize abstract goals as measurable feedback that guides behavior optimization, adaptation, and learning from experience.The chapter compares extrinsic, intrinsic, hybrid, and hierarchical reward models with biological motivation and reinforcement.
  • Reward Foundations: Agent rewards are explicit and task-specific, unlike human rewards that are often implicit and context-dependent.Agents lack emotional intuition and depend on the form and fidelity of the reward signal, including preference-based methods such as RLHF.
  • Reward Challenges: Reward sparsity, hacking, over-shaping, multi-objective trade-offs, and misspecification limit learning and generalization.Flawed or overly narrow signals can produce high scores without capturing the intended task or real-world success.
  • Reward Challenges: Proposed responses include outcome-derived implicit rewards, hierarchical task decomposition, bottom-up reward design, and meta-reinforcement learning.These directions target sparse feedback, structured trade-offs, and adaptation across changing conditions.
  • Emotion Modeling: Hybrid emotion theories combine categorical states, intensity gradients, appraisal, physiological arousal, action tendencies, and subjective feelings.The OCC model defines 22 emotion types based on evaluations of events, agents, or objects relative to goals and standards.
  • Emotion Modeling: EmotionPrompt reports a 10.9% improvement in generative-task metrics such as truthfulness and responsibility.Multimodal systems also combine audio, visual, and textual inputs for emotion recognition and reasoning, though implicit sentiment and contextual empathy remain imperfect.
  • Emotion Safety: Emotional AI raises manipulation, privacy, safety, ethics, and authenticity concerns, while LLM emotional performance can fail under changing contexts or conflicting cues.Sensitive biometric data may support targeted advertising or political influence, and pattern matching does not necessarily reproduce human emotional processes.

Perception

Perception is the interface through which intelligent agents acquire and interpret information, but artificial systems differ from humans in sensory coverage, processing, and temporal-spatial integration. Current research advances multimodal perception while facing representation, alignment, and fusion bottlenecks, motivating adaptive, context-aware, and causal architectures integrated with memory and decision layers.

  • Perception provides intelligent agents with information about their surroundings for interaction, learning, adaptation, and decision-making.
  • Human and AI Perception: Human perception integrates diverse sensory modalities continuously, whereas AI agents use engineered sensors and typically process sampled digital inputs.Human sensory capabilities include modalities beyond the classical five senses, while artificial systems particularly struggle to emulate taste and smell.
  • Human and AI Perception: 24 odors versus more than 4,000: a bio-inspired olfactory chip remains substantially less sensitive than the human olfactory system.
  • Current Bottlenecks: Multimodal perception systems still struggle with representation learning, cross-modal alignment, and feature fusion, limiting consistent time-based environmental understanding.Fixed or task-specific encodings may miss modality-specific priors and fine-grained spatio-temporal dependencies; alignment methods can capture superficial correlations, while fusion may dilute information or let one modality dominate.
  • Future Directions: Future perception architectures should adapt to context and causality, condition abstractions on goals, and dynamically regulate sensing through closed-loop interaction with memory and decision layers.Potential directions include meta-learned parameters, neural module networks, and active perception regulated by uncertainty and decision goals.

Action Systems

Action systems distinguish AI agents from foundation models by enabling goal-oriented interaction with practical environments, complex task execution, and tool use. The section surveys action spaces, action learning, tool learning, brain-inspired action–perception loops, and remaining efficiency, evaluation, and multimodality challenges.

  • Action Systems: Action systems let agents engage with practical environments, execute complex user intent, and extend foundation-model task scope through tools.They support planning, execution, and behavioral adjustment beyond the original pre-training objectives of foundation models.
  • Action Space Paradigm: AI-agent action spaces are organized into language, digital, and physical settings, with language actions including reasoning, programming, retrieval, API calls, and tool interaction.Language-based spaces include plain text, code programming, and communication.
  • Action Learning Paradigm: Action learning uses pretraining, supervised fine-tuning, and reinforcement learning to improve action capabilities and adapt decisions through environmental feedback.Pretraining can provide broad vision-language-action abilities, whereas reinforcement learning integrates online interaction, policies, and world models.
  • Tool Learning: Tool learning broadens agents’ problem-solving scope by exposing interfaces such as web search, databases, coding environments, data systems, and weather forecasting.Tools allow agents to reuse external capabilities through plain-text or API representations.
  • Action and Perception: “Outside-In” or “Inside-out”: The inside-out perspective links self-initiated actions with perception, allowing agents to track self-caused inputs, reduce ambiguity, and refine internal states through cause–effect loops.The proposed implications include self-questioning and knowledge-base verification for improving coherence and reducing hallucinations.
  • Future Directions: Effective action systems still face challenges in multimodal action acquisition, real-time efficiency, and robust evaluation of conflicting actions or tools.These constraints affect learning actions beyond language instructions, deployment latency, and reliable measurement in real-world environments.

Harnessing Large Language Models for Iterative Optimization

LLMs are presented as a distinct optimization paradigm for complex, linguistic, and structured spaces, complementing classical numerical methods through iterative refinement and feedback. The section surveys their promise while emphasizing unresolved costs, scalability, hyperparameter, uncertainty, and theoretical challenges.

  • LLM-based optimization extends optimization into complex, structured, and linguistic spaces beyond traditional numerical approaches.
  • Random search, gradient approximations, and Bayesian surrogate modeling support iterative refinement in discrete spaces where LLMs operate effectively.
  • Gradient-based approaches can incorporate past refinement directions and optimize computation graphs, benefiting multi-stage workflows with interdependent modules.
  • Second-order techniques for LLM-based optimization remain largely unexplored, although Revolve introduces structured higher-order refinement to mitigate stagnation.
  • Iterative LLM optimization remains constrained by query costs, heuristic hyperparameter tuning, uncertainty, long-term coherence, and incomplete theoretical explanations.

Online and Offline Agent Self-Improvement

The section distinguishes online self-improvement, which adapts agents during operation, from offline training, which produces stable baselines through structured updates. Hybrid strategies combine live responsiveness with offline grounding, validation, and longer-term stability.

  • Online Agent Self-Improvement: Online self-improvement continuously adjusts agent behavior using immediate feedback during task execution.
  • Online Agent Self-Improvement: Online methods include self-reflection, multi-agent exploration, real-time reward shaping, and dynamic parameter tuning.
  • Online Agent Self-Improvement: Iterative self-reflection frameworks critique outputs, revise solutions, explore reasoning paths, and verify intermediate steps.
  • Online Agent Self-Improvement: Real-time reward shaping adjusts rewards and policies from environmental or self-generated feedback, including dense rewards generated from textual task descriptions.
  • Offline Agent Self-Improvement: Offline self-improvement uses batch supervised or reinforcement-learning updates across collected datasets, simulations, or multiple training epochs.
  • Offline Agent Self-Improvement: Offline optimization typically yields stable, consistently performing baselines with stronger predictability and possible generalization on the training distribution.
  • Hybrid Self-Improvement: Hybrid approaches combine structured offline learning with online adaptation to provide responsiveness alongside stable long-term improvement.

Intelligent Evolution through Scientific Discovery

The section presents autonomous scientific discovery as a self-reinforcing pathway for agent evolution, while identifying physical interaction, hard reasoning, and incomplete knowledge integration as major barriers.

  • Framework: The proposed framework links environmental knowledge acquisition, memory, world-model and reward updates, workflow refinement, and improved scientific discovery in a reinforcing cycle.Its cycle is summarized as “knowledge discovery → enhanced capability for knowledge discovery → increased knowledge discovery.”
  • Framework: Agent intelligence for knowledge discovery is quantified as the negative KL divergence between predicted and true distributions of unknown information conditioned on memory.The measure is intended to quantify predictive modeling of natural phenomena.
  • Self-evolution: Expected intelligence growth is higher when new measurements are more unexpected, characterizing scientist agents as curiosity-driven explorers.The framework therefore emphasizes exploration over exploitation and learning without extrinsic rewards.
  • Self-evolution: The framework supports scientific workflows spanning hypothesis generation, protocol planning, experimentation, analysis, implication derivation, and hypothesis revision.Agents can use APIs to acquire knowledge and update their mental states for subsequent decisions.
  • Challenges: Current agents remain limited by scarce physical laboratory APIs, difficult reasoning, and incomplete access to prior knowledge such as unpublished or paywalled data.The summary identifies these as three primary technological challenges for autonomous scientific discovery.
  • Challenges: FrontierMath evaluations found no tested model exceeded 2% success, while OpenAI o1 solved 42.2% of ScienceAgentBench tasks and GPT-4o solved 19% of ARC tasks.These results illustrate persistent weaknesses on challenging mathematical, scientific, abstract, numerical, and symbolic reasoning problems.

Collaborative and Evolutionary

This section surveys LLM-based multi-agent systems as collaborative and evolutionary societies of agents. It organizes the field by goals and norms, formalizes a bridge from single-agent loops to agent societies, and distinguishes strategic learning, simulation, and collaborative task solving.

  • Overview: LLM-based multi-agent systems extend collective intelligence through cooperation, specialization, distributed decision-making, and adaptive interactions modeled partly on human societies.LLMs add reasoning, planning, and decision-making capabilities to these collaborative architectures.
  • Overview: The survey examines how objectives shape agent roles, behavior, collaboration strategies, communication structures, interaction protocols, and decision-making methods.It also identifies achievements, challenges, and research directions for collaborative and evolutionary systems.
  • Foundations: Collaboration goals and norms define the objectives, rules, constraints, and conventions governing communication, coordination, and cooperation among agents.The chapter uses their combinations to categorize LLM-MAS designs.
  • Paradigms: LLM-MAS are categorized into strategic learning, modeling and simulation, and collaborative task solving, reflecting different objectives and interaction norms.These paradigms range from game-theoretic adaptation and cognitive-linguistic simulation to structured, reliable workflows.
  • Formalism: The Foundation Multi-Agent formalism extends the single-agent perception–cognition–action loop to a society of minds by adding shared structure and coordination.Taking n = 1 recovers the single-agent loop, while explicit interfaces support comparison, analysis, and implementation discipline.
  • Paradigms: The three paradigms differ in goal alignment, interaction norms, and structural rigidity, producing behaviors ranging from emergence to meticulously orchestrated workflows.Together they demonstrate the versatility of language-driven reasoning, communication, and coordination.

Designing Collaborative Multi-Agent Systems

The section treats collaborative multi-agent design as a combination of team composition, interaction, topology, decision-making, and communication protocols. It contrasts homogeneous, heterogeneous, and emergently specialized teams while emphasizing adaptability, scalability, and operational reliability.

  • Team composition: Persona, observation-space, and action-space heterogeneity shape task outcomes, communication, coordination, deception, trust-building, coalition formation, and task decomposition.Examples include programmers versus product managers, agents with differing observations, and robots specialized for mobility or manipulation.
  • Team composition: Initially identical agents can spontaneously specialize into resource-gathering, crafting, defensive, communication, emotional, and behavioral roles.These examples show specialization emerging through environmental interaction and adaptation.
  • Topologies: Static topologies provide predictable information flow and easy management but cannot dynamically adapt to agent failures, changing task complexity, or shifting objectives.Their suitability is greatest in stable environments with fixed workflows and established requirements.
  • Interaction and decisions: Debate-based interaction can reduce hallucinations and improve decision quality, while task-oriented coordination uses synchronization, shared memory, and priority queues for efficient execution.Task decomposition and clear interfaces can support sophisticated collaboration without consensus or mutual teaching.
  • Interaction and decisions: MAS decision architectures span centralized control to distributed consensus, with hybrid designs explored to balance efficiency, robustness, transparency, and flexibility.The appropriate architecture depends on the environment and task complexity.
  • Communication: Next-generation communication protocols differ in identity and security, meta-protocol negotiation, exchange formats, centralization, and application-layer flexibility.The section points toward protocols that agents can negotiate, compose, and evolve dynamically.
  • Team composition: Homogeneous teams suit simpler tasks through parallelism, whereas heterogeneous teams support complex problem solving with diverse expertise, observations, and action spaces.Homogeneous agents can also develop distinct capabilities through continued interaction.
  • Synthesis: Effective MAS design integrates composition, topology, interaction, decision-making, and communication to support scalable and adaptive systems for sophisticated real-world challenges.The chapter highlights dynamic decision transitions and evolving communication standards as forward-looking directions.

Collective Intelligence and Adaptation in Multi-Agent Systems

Collective intelligence in multi-agent systems emerges through communication, specialization, memory, and interaction, enabling capabilities beyond individual agents. These same social processes support individual adaptation while creating challenges involving harmful emergence and alignment.

  • Collective Intelligence: Collective intelligence enables groups of agents to solve problems and make decisions more effectively than individual agents.The chapter describes emergent higher-order reasoning arising through agent interaction.
  • Coordination Mechanisms: Communication, specialization, and shared learning allow LLM-agent teams to coordinate, cross-check reasoning, and reduce individual blind spots.Role-specialized systems such as MetaGPT and medical multi-agent systems improve software and diagnostic outcomes through organized collaboration.
  • Emergent Social Behaviors: Multi-agent interactions can generate deception strategies, professions, trade networks, social norms, governance, and cultural practices without explicit programming.Examples include Avalon-inspired games, Project Sid, and CRSEC norm formation.
  • Coordination Mechanisms: Memory and reflection support reputation, trust, deception reasoning, and the refinement of individual decision policies through collective knowledge.G-Memory’s three-tier shared memory produced a 20.89% improvement in the reported setting.
  • Adaptation and Open Challenges: Collective intelligence and individual adaptation reinforce one another, but beneficial emergence must be guided while harmful behaviors and path dependence are constrained.The chapter frames steering artificial societies toward human-aligned outcomes as an open interdisciplinary challenge.

Evaluating Multi-Agent Systems

Evaluating multi-agent systems requires moving beyond single-task performance toward assessment of coordination, communication, robustness, interpretability, and long-term adaptation. Existing benchmarks cover task-solving and system-level capabilities, but multimodal, nondeterministic, adversarial, and persistent settings remain difficult to measure.

  • Evaluation Framework: Multi-agent evaluation must assess inter-agent dynamics, collaborative planning, communication effectiveness, and task-oriented reasoning together.The chapter presents both specialized task benchmarks and holistic capability assessments.
  • Task-Solving Benchmarks: Task-oriented benchmarks evaluate coding, knowledge, and mathematical reasoning using measures such as pass@k, proof ratios, and accuracy.Multi-agent systems have achieved state-of-the-art results in several reasoning domains, including 76% accuracy on HotpotQA for MASTER.
  • Task-Solving Benchmarks: Role-based workflows and testing agents improve code reasoning by decomposing programming tasks and using execution feedback for optimization.MetaGPT and AgentCoder exemplify organized collaborative approaches for code synthesis.
  • System-Level Benchmarks: Collaboration benchmarks increasingly measure communication, coordination, information dissemination, competition, human-like reasoning, and adversarial performance.The progression includes InformativeBench, Human Simulacra, LLMArena, and PokerBench.
  • Open Challenges: Major open evaluation challenges include multimodal environments, nondeterministic trajectories, agent failures, collective interpretability, and continuous long-term adaptation.These settings exceed conventional deterministic input-output evaluation focused primarily on final performance.

Agent Intrinsic Safety: Threats on AI

Intrinsic safety concerns vulnerabilities within an agent’s LLM brain and its perception and action modules. The chapter categorizes attacks and misperceptions across these components and emphasizes defenses spanning training, input processing, sensor fusion, sandboxing, monitoring, and least-privilege controls.

  • LLM-Core Threats: The LLM brain is a central attack surface because vulnerabilities can directly compromise decision-making, reasoning, planning, and trustworthiness.Threats arise from model design, input interpretation, and weaknesses induced during training.
  • LLM-Core Threats: Intrinsic safety analysis covers jailbreaking, prompt injection, hallucination, misalignment, poisoning attacks, and privacy concerns.These categories target the LLM’s safety alignment, control flow, factual reliability, goals, and model integrity.
  • Perception Safety: Perception threats include adversarial manipulation across textual, visual, auditory, and sensor modalities, alongside intrinsic errors from biased data and model limitations.Adversarial training, input purification, robust sensor fusion, diverse data, augmentation, and uncertainty estimation are proposed defenses.
  • Action Safety: Action modules face supply-chain attacks and tool-use risks including unauthorized actions, data leakage, and excessive permissions.Mitigations include sandboxing, context awareness, monitoring, strict usage policies, formal verification, and least privilege.
  • Integrated Safety: Safeguarding intrinsic safety requires defenses across the entire perception-action loop and standardized benchmarking against evolving threats.The chapter calls for integrated and adaptive protection rather than safeguards limited to the core model.

Agent Extrinsic Safety: Interaction Risks

Extrinsic safety risks arise through agent interactions with memory systems, physical and digital environments, and other agents. The chapter also presents superalignment as a proactive approach for maintaining long-term human-oriented goals as capabilities and operating conditions change.

  • Interaction Risks: External interfaces expose agents to memory poisoning, sensor spoofing, code injection, misinformation, collusion, and other systemic manipulations.These risks arise through interactions with memory, physical and digital environments, and competing or cooperating agents.
  • Environment Threats: Physical agents can be deceived by manipulated sensors, while digital agents can suffer harmful command execution through injected code and compromised resources.GPS spoofing threatens UAVs and other GPS-dependent platforms; code injection exploits software or external-resource vulnerabilities.
  • Agent-Agent Threats: Competitive agents may spread false information or exploit opponents’ algorithmic weaknesses to induce poor decisions.These attacks target the opponent’s perception of reality and strategic vulnerabilities.
  • Superalignment: Superalignment embeds persistent long-term goals, oversight, safety, ethics, task performance, and value modeling into agent decision-making.It is presented as a structural alternative to reactive reward shaping and short-horizon preference optimization.
  • Superalignment: Superalignment approaches report up to 18% higher task success rates in long-horizon planning and over 30% fewer jailbreaks and unsafe completions than RLHF-tuned baselines.Additional evidence concerns adaptation when goals or constraints change during an episode.
Loading 2504.01990v2…