PaperlayerView exact revision on arXiv

Source-linked AI summary

Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems

Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, Zhaoyang Yu, Haochen Shi, Boyan Li, Dekun Wu, Fengwei Teng, Xiaojun Jia, Jiawei Xu, Jinyu Xiang, Yizhang Lin, Tianming Liu, Tongliang Liu, Yu Su, Huan Sun, Glen Berseth, Jianyun Nie, Ian Foster, Logan Ward, Qingyun Wu, Yu Gu, Mingchen Zhuge, Xinbing Liang, Xiangru Tang, Haohan Wang, Jiaxuan You, Chi Wang, Jian Pei, Qiang Yang, Xiaoliang Qi, Chenglin Wu

arXiv:2504.01990v2cs.AI

TL;DR

LLM-based agents still lack comprehensive capabilities for complex planning, reliable long-term memory, and autonomous real-world action. This book surveys brain-inspired modular architectures, adaptive evolution, multi-agent systems, and safety, concluding that world-dynamics understanding is a key bottleneck for reasoning.

  • Problem

    LLM-based agents lack comprehensive support for complex planning, reliable long-term memory, and autonomous real-world action.

    Source: Preface · p. 3

  • Method

    The book synthesizes brain-inspired modular architectures with mechanisms for adaptive evolution, multi-agent collaboration, and safe deployment.

    Source: Core Components of Intelligent Agents · p. 39, §2.1.4 · p. 50

  • Results

    The survey identifies world-dynamics understanding as a key bottleneck limiting agents’ reasoning capabilities.

    Source: §2.3 · p. 63

  • Takeaways & Limitations

    Improving Foundation Agents may require jointly enhancing logical reasoning and world modeling rather than optimizing reasoning structure alone.

    Source: §2.3 · p. 63

    The brain-inspired mapping leaves arousal and global attention states largely unaddressed in current AI systems.

    Source: §1.2.1 · p. 25

Abstract

from arXiv · show

The advent of large language models (LLMs) has catalyzed a transformative shift in artificial intelligence, paving the way for advanced intelligent agents capable of sophisticated reasoning, robust perception, and versatile action across diverse domains. As these agents increasingly drive AI research and practical applications, their design, evaluation, and continuous improvement present intricate, multifaceted challenges. This book provides a comprehensive overview, framing intelligent agents within modular, brain-inspired architectures that integrate principles from cognitive science, neuroscience, and computational research. We structure our exploration into four interconnected parts. First, we systematically investigate the modular foundation of intelligent agents, systematically mapping their cognitive, perceptual, and operational modules onto analogous human brain functionalities and elucidating core components such as memory, world modeling, reward processing, goal, and emotion. Second, we discuss self-enhancement and adaptive evolution mechanisms, exploring how agents autonomously refine their capabilities, adapt to dynamic environments, and achieve continual learning through automated optimization paradigms. Third, we examine multi-agent systems, investigating the collective intelligence emerging from agent interactions, cooperation, and societal structures. Finally, we address the critical imperative of building safe and beneficial AI systems, emphasizing intrinsic and extrinsic security threats, ethical alignment, robustness, and practical mitigation strategies necessary for trustworthy real-world deployment. By synthesizing modular AI architectures with insights from different disciplines, this survey identifies key research challenges and opportunities, encouraging innovations that harmonize technological advancement with meaningful societal benefit.

18.1 Perception · 21 Concluding Remarks and Future Outlook · Introduction

This section examines brain-inspired foundation agents that perceive, reason, learn, adapt, and act across diverse embodiments and environments. It connects modular architectures and brain-based perspectives with challenges in cognition, continual learning, self-enhancement, collaboration, and safe, human-centered deployment.

18.1 Perception

  • The safety material scopes perception threats around adversarial attacks and misperception, alongside action threats involving supply chains and tool usage.The listed material also identifies agent-memory, agent-environment, and agent-agent interaction threats as subsequent safety categories.Source: §18.1 · p. 11

Notation

  • The notation section provides a convenience summary of symbols used throughout the book, with detailed definitions located in referenced sections.Its entries cover agent, environment, workflow, learning, alignment, prompt-injection, attention, and hallucination notation.Source: Notation · p. 13

1.1 The Rise and Development of AI Agents

  • The agent paradigm evolved from symbolic problem-solving and reactive robotics into a framework integrating perception, reasoning, and autonomous action in dynamic environments.Brooks’s behavior-based robotics showed that parallel, modular, sensory-motor behaviors could produce robust actions without detailed internal world models.Source: §1.1 · pp. 16–17
  • LLM-powered agents combine broad language-based reasoning, environmental interaction, tool use, and few-shot or zero-shot adaptability to address complex, long-horizon tasks.They can query databases, execute actions through function-calling APIs, and mediate between human intent and machine execution.Source: §1.1 · p. 17

1.2 A Parallel Comparison between Human Brain and AI Agents

  • Human cognition differs from LLM-powered agents in underlying hardware, consciousness, learning, creativity, and energy efficiency, making the comparison informative but necessarily high-level.The comparison is presented as a snapshot rather than an exhaustive account of either human intelligence or AI-agent capabilities.Source: cognition · pp. 17–18

cognition.

  • Human brains remain more energy-efficient and adaptable, while LLM agents rely mainly on computationally intensive offline training and statistical recombination rather than lived, emotional creativity.Proposed directions include lifelong learning, personalized adaptation, interactive fine-tuning, contextual understanding, simulated emotion, and experiential memory.Source: cognition · p. 18

1.2.1 Brain Functionalities and AI Parallels

  • Brain-inspired analysis links major regions and cognitive functions to AI capabilities, while treating cognition as distributed across interacting networks rather than rigidly localized.The framework highlights reasoning, memory, planning, perception, decision-making, motivation, emotion, and motor skills, and classifies them as well-developed (L1), partially developed (L2), or underexplored (L3).Source: §1.2.1 · pp. 18–19
  • Current AI partially reproduces executive and perceptual functions, but flexible planning, open-ended decision-making, adaptive attention, multisensory integration, and general spatial understanding remain limited.Planning and reasoning, decision-making, multisensory integration, spatial mapping, attention, and sensorimotor coordination are generally L2, while touch sensing is L3; frontal inhibitory control and strategy shifting remain underexplored (L3).Source: §1.2.1 · pp. 20–22
  • Visual recognition and many language functions are well-developed (L1), whereas visual reasoning, grounded scene understanding, auditory scene analysis, semantic understanding, and episodic lifelong learning remain incomplete.AI vision can match or exceed human performance on object recognition, while language comprehension and production are highly capable in narrow settings; episodic memory and lifelong learning are classified L3 because of unstable continual learning and catastrophic forgetting.Source: §1.2.1 · pp. 22–23
  • Cerebellum-inspired capabilities expose gaps in real-time motor coordination, timing, prediction, and adaptive error correction, motivating architectures that separate supervised, reinforcement, and unsupervised learning paradigms.Robotic motor learning and online feedback control are mostly task-specific or L2, while general cognitive timing and prediction are relatively underexplored (L3).Source: §1.2.1 · p. 24
  • Brainstem functions highlight that human-like intelligence is embodied: reflexive robotic responses can be narrow L1 successes, but autonomic regulation and global arousal or attention states are largely unaddressed (L3).AI generally lacks physiological self-maintenance, energy regulation, and alertness modulation, underscoring a conceptual gap between embodied biological intelligence and largely disembodied systems.Source: §1.2.1 · p. 25
  • Brain–AI comparisons identify major gaps in thalamic routing, intrinsic drives, emotion, episodic memory, and rich subcortical coordination, motivating AI that selectively adopts useful cognitive principles while prioritizing safety, interpretability, oversight, and human empowerment.The section cautions that intrinsic motivation may be dangerous and frames AI as augmenting rather than replacing human abilities within a transparent, controllable, human-centered society.Source: §1.2.1 · pp. 26–27

1.3 Foundation Agents: A Modular and Brain-Inspired AI Agent Framework

  • The proposed brain-inspired architecture decomposes agents into perception, cognition, and action, with cognition integrating memory, world models, emotion, goals, rewards, learning, reasoning, planning, and decision-making.The framework treats biological mechanisms as inspiration rather than direct replication and incorporates attention for selective filtering plus emotional, rational, and reward-driven processes.Source: §1.3 · p. 28, §1.3.2 · pp. 31–32

1.3.1 From Language Models to AI Agents

  • LLM agents require faculties beyond language, including perception, action, memory, world modeling, motivation, and learning, connected through a recurring perception–cognition–action loop.These faculties introduce challenges in multimodal fusion and attention, safe tool use, selective retrieval, predictive planning, goal design, and continual learning without catastrophic forgetting or unsafe drift.Source: §1.3.1 · pp. 28–29

1.3.2 Core Concepts and Notations in the Agent Loop

  • Foundation Agents are defined as autonomous, adaptive systems that perceive diverse signals, continually update structured internal states, and reason about purposeful actions toward complex long-term objectives.Their capabilities include multimodal perception, dynamic cognitive adaptation, autonomous reasoning and planning, and external or internal purposeful action.Source: §1.3.2 · p. 33
  • Foundation Agents integrate perception, cognition, and action with evolving memory, world models, emotion, rewards, goals, reasoning, and internal or external actions.This brain-inspired formulation supports goal decomposition, strategic exploration, adaptive behavior, and diverse embodiments.Source: §1.3.2 · p. 33

1.3.3 Biological Inspirations

  • Biological analogies guide, rather than rigidly determine, computational modules, grounding adaptive agent behavior in cognitive principles without requiring biological exactitude.The framework draws on brain-inspired parallels while preserving implementation flexibility.Source: §1.3.3 · p. 34, §1.3.4 · p. 36

Memory:

  • The architecture models memory as short- and long-term storage, world models as predictive simulations, emotion as adaptive guidance, goals and rewards as motivational control, and reasoning as executive planning.These modules selectively retain experience, update predictions, prioritize attention, reinforce strategies, and coordinate deliberative or rapid decisions.Source: Memory · pp. 34–35

1.3.4 Connections to Existing Theories

  • The framework generalizes classical POMDPs through flexible deterministic or stochastic transitions, internalized rewards, richer decision-making, and modular mental states.Classical POMDPs remain a simplified special case, while the expanded formulation supports more varied and interpretable behaviors.Source: §1.3.4 · p. 36

1.4 Navigating This Book

  • The book positions its contribution among prior agent surveys and theories by systematically comparing biological cognition with computational frameworks to identify synergies, gaps, and innovation opportunities.It explicitly connects the design to perception–cognition–action cycles, Society of Mind, inside-out perception, active inference, and related theories.Source: §1.3.4 · pp. 35–36, §1.4 · p. 36

Navigating This Book 37

  • The book is organized into four parts covering modular agents, self-enhancement, collaborative and evolutionary systems, and safe and beneficial AI.Its safety part distinguishes intrinsic and extrinsic threats, examines attacks and defenses, and considers superalignment and capability–risk scaling.Source: Navigating This Book 37 · p. 37

Core Components of Intelligent Agents

Intelligent agents are structured minds in which perception, cognition, and action interact through a dynamic mental state. Their cognition integrates memory, world modeling, emotion, goals, reward, and learning to support adaptation and purposeful behavior.

Core Components of Intelligent Agents

  • Intelligence emerges from the interplay of perception, cognition, and action, driven by a dynamic mental state.The agent operates as a living loop in which perception feeds cognition, cognition selects action, and action reshapes perception.Source: Core Components of Intelligent Agents · p. 39
  • Perception serves as the gateway from world to mind, while action channels the mind’s influence back into the world.Together, these modules support agents that perceive their environment, adapt to change, and pursue long-term purpose.Source: Core Components of Intelligent Agents · p. 39
  • Cognition organizes thought, planning, and decisions while integrating memory, world models, emotion, goals, reward, and learning.Memory connects past and present, world models imagine futures, reward learns from outcomes, and emotion biases cognition and behavior.Source: Core Components of Intelligent Agents · p. 39
  • These biologically inspired modules provide the internal machinery underlying intelligent behavior and establish the foundation for digital agents.The framework presents each component as suitable for building digital minds while drawing inspiration from how biological systems, especially the human brain, solve related problems.Source: Core Components of Intelligent Agents · p. 39

Cognition

This section frames foundation-agent cognition as a modular, adaptive cycle linking learning, perception, reasoning, world modeling, memory, rewards, and continual environmental adaptation. It unifies reasoning through resource-aware operator–scheduler processes over imagined futures, while examining structured and latent computation, feedback, search, meta-learning, internal motivation, and the challenges of generalization, planning, and world understanding.

Cognition

  • Cognition is modeled as an integrated modular architecture in which perception, memory, reasoning, action, reward, and emotion operate over structured mental states.LLM-based agents approximate these functions through hidden representations, memory buffers, and intermediate reasoning steps, with learning and reasoning as central processes.Source: Cognition · p. 40

2.1.1 A Unified Formulation of Learning

  • Learning is unified as optimization balancing experience fidelity, internal-model parsimony, and foresight across multiple temporal horizons.The formulation incorporates prediction error, action costs, temporal mixtures, and complexity regularization, with diverse learning paradigms recovered by instantiating these terms.Source: §2.1.1 · pp. 41–42

2.1.3 Learning Space

  • Agent learning is presented as data-driven and computationally structured, while human learning integrates curiosity, emotion, neural specialization, and adaptation across multiple timescales.This contrast motivates architectures that combine computational strengths in large-scale knowledge and formal representation with more adaptive, contextual learning mechanisms.Source: §2.1.2 · p. 43, §2.1.3 · p. 44
  • Learning spans holistic and component-level changes, updating model parameters or selectively adapting memory, world models, rewards, goals, and emotional state.Full updates affect broad capabilities, whereas targeted or in-context updates preserve general abilities while adapting specific subsystems.Source: §2.1.3 · pp. 45–46

2.1.2 Learning Across Mental State Components

  • LLM agents learn through pretraining, supervised and reinforcement updates, and in-context adaptation, combining large-scale knowledge acquisition with task-specific and context-sensitive improvement.In-context learning adapts without parameter changes, while parameter updates refine capabilities such as alignment, reasoning, planning, and environmental action selection [68] [71] [116].Source: §2.1.3 · pp. 45–46

2.1.4 Learning Objective

  • Agents continuously learn across perception, reasoning, and world comprehension, enabling adaptation to complex and dynamic environments.Perceptual learning expands multimodal integration through visual, linguistic, audio, and tactile representations, including CLIP, LLaVA, CogVLM, and Qwen-Audio [80].Source: §2.1.4 · p. 47
  • Agents improve reasoning through data-driven training, reinforcement learning, and self-evolution, with high-quality reasoning paths activating latent capabilities more effectively than data quantity alone.SFT can activate reasoning without complex algorithms, while LIMO and related findings emphasize structured reasoning chains and automated selection of high-quality data.Source: §2.1.4 · pp. 48–49
  • Robust self-improvement remains constrained by reward design because outcome-only signals can reward flawed reasoning, motivating process supervision and intrinsic consistency or self-critique signals.Verifree explores intrinsic rewards to reduce reliance on costly external verifiers, while self-training effectiveness depends on initial model capability and task characteristics.Source: §2.1.4 · p. 50
  • Agents increasingly self-improve through multi-agent reinforcement learning and code-level modification, enabling collaboration, finer-grained credit assignment, autonomous capability construction, and continuous rewriting of operational mechanisms.MARFT, SPA-RL, GiGPO, and RAGEN address coordination, credit assignment, and stability, while Alita, AlphaEvolve, and the Darwin Gödel Machine extend self-improvement to protocols and code.Source: §2.1.4 · p. 50
  • World understanding develops through interaction, experience processing, reward refinement, and explicit world models that support simulation, planning, safer decisions, and autonomous experiential learning cycles.Reflexion and ExpeL manage experience collection, analysis, and application across successes and failures, making world models central to interactive learning.Source: §2.1.4 · p. 51

2.2 Reasoning

  • Reasoning transforms mental states into actions by integrating knowledge, multimodal information, and environmental feedback through structured compositions or flexible unstructured processes.Structured reasoning composes explicit steps, whereas unstructured reasoning preserves implicit, context-adaptive composition; environmental interaction updates mental states and validates outcomes.Source: §2.2 · pp. 52–53

2.2.1 A Unified Formulation of Reasoning

  • The unified reasoning objective selects policies minimizing expected future free energy while balancing utility, epistemic exploration, action costs, and computational resource constraints.The formulation recovers exploitation or curiosity through α, incorporates practical penalties such as risk and API fees, and supports bounded rationality through search cost.Source: §2.2.1 · p. 53
  • Reasoning is realized as a scheduler repeatedly selecting operators from a library to transform an initial hypothesis until termination or resource exhaustion, then extracting an executable action.The resulting trace subsumes classical search trees, stochastic rollouts, chain-of-thought generation, symbolic planning, neural program synthesis, and program-synthesis planners.Source: §2.2.1 · p. 54, Figure 2.6 · p. 54
  • A unified operator–scheduler formulation captures diverse reasoning styles within the perception–cognition–action loop of Foundation Agents.Operators can interact with world models, memory, and emotion states while sharing globally updated parameters.Source: §2.2.1 · p. 55

2.2.2 Structured Reasoning

  • Structured reasoning organizes explicit computational steps, progressing from linear sequences to trees, graphs, ensembles, iterative refinement, verification, and domain-specific frameworks.These structures improve traceability, alternative-path exploration, error correction, and task-specific performance across diverse domains.Source: §2.2.2 · p. 55, §2.2.2.1 · p. 56, §2.2.2.2 · pp. 56–57

2.2.2.1 Dynamic Reasoning Structures

  • Dynamic reasoning structures adapt paths using sequential, tree-based, or graph-based exploration, including ReAct [97], ToT, LATS, GoT, PoT [103], and DoT.These methods support action feedback, branching search, world-model simulation, non-hierarchical dependencies, and logically organized refinement.Source: §2.2.2.1 · pp. 55–56

2.2.2.2 Static Reasoning Structures

  • Static reasoning improves fixed frameworks through ensemble aggregation, progressive feedback, self-refinement, and verification against errors, hallucinations, or external tool feedback.Self-Consistency, Self-Refine, Reflexion, Chain-of-Verification, and Critic exemplify these mechanisms.Source: §2.2.2.2 · pp. 56–57

2.2.2.3 Domain-Specific Reasoning Frameworks

  • Domain-specific frameworks tailor reasoning to mathematics, physics, and education through multiple solution paths, knowledge-augmented formulas, checklists, and pedagogical error identification.The broader evolution runs from linear and graph structures to specialized frameworks that improve reasoning across tasks and domains.Source: §2.2.2.3 · pp. 57–58

2.2.3 Unstructured Reasoning

  • Unstructured reasoning uses flexible implicit or language-space computation, while prompting methods guide inference through demonstrations, decomposition, abstraction, reformulation, and knowledge integration.Step-Back Prompting reports improvements of 7-27% across physics, chemistry, and multi-hop reasoning benchmarks.Source: §2.2.3 · p. 58, §2.2.3.1 · pp. 58–59

2.2.3.2 Reasoning Models

  • Specialized reasoning models such as DeepSeek’s R1 [116], Claude 3.7 Sonnet, and OpenAI’s o series models [117] exceed general-purpose models on complex inference, mathematics, and multi-step problems.Their gains derive from reasoning-focused data, architectures, objectives, human feedback, and reinforcement learning.Source: §2.2.3.2 · p. 59

2.2.3.3 Implicit Reasoning

  • Implicit methods reduce generated tokens while preserving reasoning: Quiet-STaR [118] improves mathematics from 5.9%→10.9% and commonsense reasoning from 36.3%→47.2%, whereas Coconut [119] reasons in continuous latent space.System-1.5 Reasoning [192] combines explicit deliberation with latent parallel exploration, but implicit reasoning can overfit patterns and fail to generalize.Source: §2.2.3.3 · pp. 59–60

2.3 Summary and Discussion

  • Learning updates mental states while reasoning uses them for decisions, creating an experience-driven spiral that continuously enhances cognitive capabilities.Richer experiential data improves learning effectiveness, while better mental-state representations support more efficient reasoning.Source: §2.3 · p. 62
  • Agents can improve through prompts, workflows, examples, memory, action spaces, logic, data acquisition, and training, motivating meta-learning over learning forms.Foundation Agents should learn how to select or construct appropriate architectures for different scenarios and continuously improve them.Source: §2.3 · p. 62
  • Agents lack the internal reward systems that drive human exploration, so solving artificial internal motivation is necessary for active learning and spontaneous improvement.External goals alone produce behavior that is more passive and reactive than intrinsically motivated biological behavior.Source: §2.3 · p. 62
  • Shen et al. [172] and ActRe [175] show that acquiring or embedding environmental dynamics understanding improves reasoning, suggesting world understanding is a key capability bottleneck.Current reasoning work emphasizes mathematical and logical domains, while comparatively fewer studies improve agents’ understanding of external world dynamics.Source: §2.3 · pp. 62–63
  • Future Foundation Agents should jointly improve logical reasoning, adaptive world understanding, sophisticated learning, and internal drives to autonomously evolve in changing environments.This requires balancing passive external guidance with active internal motivation and examining the relationship between reasoning and world modeling.Source: §2.3 · p. 63

Memory

This section examines memory as a layered, lifecycle-based foundation for humans and intelligent agents, spanning transient retention, active processing, long-term storage, selective acquisition and forgetting, retrieval, and use in reasoning and generation. It also considers memory architectures and their integration with world models and other cognitive functions, while identifying trade-offs and gaps involving adaptability, efficiency, stability, transparency, integration, and generalization.

3.1 Overview of Human Memory

  • Human memory comprises interacting subsystems that differ in timescale, representation, and conscious accessibility, supporting learning, behavioral adaptation, perception, skills, and future projection.The overview frames memory as a constellation rather than a monolithic faculty, with encoding, storage, and retrieval organized across multiple systems.Source: §3.1 · p. 64

3.1.1 Types of Human Memory

  • Human memory is hierarchically organized into sensory, short-term or working, and long-term stores, with declarative and non-declarative forms within long-term memory.Sensory memory briefly preserves raw inputs; short-term or working memory maintains and manipates limited information; long-term memory supports durable knowledge, skills, and experiences but remains affected by decay, interference, and retrieval cues.Source: §3.1.1 · pp. 65–66, Figure 3.1 · p. 65

3.1.2 Models of Human Memory

  • Influential human-memory models emphasize staged storage, active manipulation, integrated cognitive and action systems, global broadcasting, or symbolic–subsymbolic control.These include the multi-store model, working-memory models [246, 247, 248], Tulving’s SPI model [249], GWT and IDA/LIDA, and ACT-R.Source: §3.1.2 · pp. 66–69

3.2 From Human Memory to Agent Memory

  • LLM-agent memory supplies context retention, experience-based learning, and coherent long-horizon behavior, requiring internalized reasoning, interim steps, evolving objectives, and dialogue history beyond external knowledge bases.Agent approaches have progressed from appending conversation history to vector retrieval and selective incorporation of reasoning chains [257].Source: §3.2 · pp. 69–70
  • Current agent memory remains less associative, adaptive, and stable than human memory, with weaknesses in ambiguity handling, consolidation, selective forgetting, metacognitive oversight, integration, and knowledge transfer.Digital systems separate storage from computation and often rely on precise matching or vector similarity, whereas human memory extracts salient information from noisy data and supports fuzzy associations.Source: §3.2 · pp. 69–70

3.3 Representation of Agent Memory

  • Agent memory architectures organize sensory, short-term, and long-term layers into a cycle that converts observations into actionable representations, supports state management and replay, and consolidates knowledge across tasks.This layered design links perception, interpretation, and response while balancing immediate responsiveness with continuous learning and broader contextual continuity.Source: §3.3 · p. 70

3.3.1 Sensory Memory

  • Agent sensory memory encodes and attentively filters multimodal inputs before transient retention, but converting already abstracted observations into representations suitable for reasoning can lose or distort vital raw sensory information.The process is described as perceptual encoding, attentional selection, and transient retention; systems such as RecAgent [259], Jarvis-1, VideoAgent, and WorldGPT address text or multimodal perception.Source: §3.3.1 · pp. 72–73
  • Memory spans sensory retention, short-term processing, and long-term storage, enabling agents to preserve information across immediate responses, ongoing tasks, and future adaptation.Sensory memory temporarily retains selected observations; short-term memory bridges sensory and long-term memory; long-term memory supports cumulative learning and cross-task adaptability.Source: §3.3.1 · p. 73, §3.3.2 · p. 73, §3.3.3 · p. 75

3.3.2 Short-Term Memory

  • Short-term memory acts as a transient workspace that integrates recent observations, task demands, goals, plans, and reasoning for real-time decision-making and adaptive behavior.Context memory extends or structures LLM context, while working memory retrieves and integrates external knowledge, feedback, trajectories, and environmental state for dialogue, GUI, game, and robotic tasks.Source: §3.3.2 · pp. 73–74, Figure 3.8 · p. 74

3.3.3 Long-Term Memory

  • Long-term memory combines semantic and episodic recollection with procedural memory and priming to support knowledge accumulation, context-aware responses, skill reuse, and generalization.Semantic memory stores facts and concepts, episodic memory records events and interaction histories, while procedural formats such as code and reusable routines are retrieved for task-specific planning.Source: §3.3.3 · pp. 75–76, Figure 3.9 · p. 75

3.4 The Memory Lifecycle

  • The memory lifecycle comprises retention through acquisition, encoding, and derivation, followed by retrieval through matching, neural memory networks, and utilization to guide real-time behavior.This dual-process view governs how information is captured, structured, refined, accessed, and applied for learning, contextual awareness, and decision-making.Source: §3.4 · pp. 76–77, Figure 3.10 · p. 76
  • The lifecycle is framed as a latent optimization problem that approximates a minimal sufficient memory state while balancing semantic divergence, storage cost, retrieval cost, and fidelity–efficiency trade-offs.The divergence term Lmem evaluates how useful, sufficient, and task-aligned the current memory is, while λstore and λretr regulate storage and retrieval efficiency.Source: §3.4 · p. 77

3.4.1 Memory Acquisition

  • Memory acquisition filters and compresses high-volume perceptual inputs into representative sketches, preserving task-relevant features while discarding noise and redundancy.It is treated as the agent’s first judgment about what to remember, using compression and experience consolidation to support efficiency and later generalization.Source: §3.4.1 · pp. 77–78

3.4.2 Memory Encoding

  • Memory encoding transforms acquired sketches into structured latent representations using selective attention and multimodal fusion aligned with downstream goals and retrieval needs.These mechanisms focus computation on relevant input regions or tokens and align visual, auditory, textual, and plan-related information into unified memory representations.Source: §3.4.2 · pp. 78–80

3.4.3 Memory Derivation

  • Memory derivation reorganizes stored content through reflection, summarization, knowledge distillation, and selective forgetting to reduce redundancy and staleness while improving abstraction, efficiency, and generalization.The process is guided by memory utility and structure rather than explicit targets, and can correct failed trajectories, condense long histories, or transfer reasoning and decisions into smaller models.Source: §3.4.3 · p. 80
  • Selective forgetting removes or down-weights irrelevant, redundant, outdated, or infrequently used memories through time-, frequency-, and relevance-based mechanisms, supporting efficient updates.MemoryBank models time decay and spacing effects, while Lyfe Agent summarizes clustered memories before removing older, highly similar ones for real-time social interactions.Source: §3.4.3 · p. 81

3.4.4 Memory Retrieval and Matching

  • Context-aware retrieval must unify heterogeneous memories, match them to current states and goals, operate efficiently in real time, and update them as experiences and knowledge evolve.A comprehensive design uses shared representations and hybrid indexes, contextual attention, task-oriented dynamic routing with repeated retrieval, and forgetting and updating based on time and access frequency.Source: §3.4.4 · pp. 81–82

3.4.5 Neural Memory Networks

  • Neural memory networks integrate experiences into model parameters, enabling embedded access, reduced explicit management, and higher-order reasoning, but they face bounded capacity, reduced controllability, and catastrophic forgetting.Parametric memory trades fluid integration and richer associations for scalability and editability challenges, motivating more stable read-write mechanisms and hybrid memory architectures.Source: §3.4.5 · pp. 83–84, §3.5 · p. 87

3.4.6 Memory Utilization

  • Memory utilization combines retrieved information with generation, long-context modeling, and hallucination mitigation to support more factual, reliable reasoning, planning, action, and dialogue.RAG grounds outputs in external knowledge, long-context methods expand or compress usable context, and fact-checking, uncertainty estimation, knowledge-based decoding, and expert memory subnetworks reduce errors.Source: §3.4.6 · pp. 85–86

3.5 Summary and Discussion

  • Memory systems must integrate with perception, planning, reasoning, action selection, and predictive world models, while addressing abstraction, multimodal integration, bias, uncertainty, and continuous adaptation.The section identifies a gap between human associative, consolidating, and abstracting memory and agent memory that often depends on precise matching and struggles with ambiguity.Source: §3.5 · p. 86

World Model

This section examines world models as predictive mechanisms for reasoning about future states and situates them within agent architectures. It compares implicit, explicit, simulator-based, and hybrid or instruction-driven paradigms, including their mechanisms, trade-offs, and interactions with other agent components.

4.1 Mental Models as Human World Representations

  • Human mental models integrate perception, memory, and reasoning to predict outcomes, plan actions, adapt to errors, and operate across multiple temporal and spatial scales.They are predictive, integrative, adaptive, and multi-scale rather than static stores of facts.Source: §4.1 · p. 89

4.2 From Human Mental Models to AI World Models

  • AI world models operationalize the mental-model idea by internally simulating environments for imagination, predictive reasoning, sample-efficient learning, and adaptation without executing real actions.Examples include latent generative models, explicit forward models, and simulator-grounded systems.Source: §4.2 · p. 90

4.3 Paradigms of AI World Models

  • The section classifies AI world models as implicit, explicit, simulator-based, or hybrid/instruction-driven according to observation reconstruction, internal versus external mechanisms, and system complexity.Implicit models plan in latent spaces, explicit models reconstruct future observations, simulator-based systems query external environments, and hybrids combine learned, symbolic, textual, or interactive components.Source: §4.3 · p. 90, §4.3.1 · p. 92

4.3.6 Comparative Summary of Paradigms

  • Across the paradigms, world-model methods vary in internal or external form, complexity, rollout and planning mechanisms, and training objectives, creating task-dependent trade-offs rather than a single universally superior design.The comparative summary organizes representative methods by form, complexity, and paradigm.Source: Table 4.1 · p. 91, Table 4.2 · p. 95, §4.3.6 · p. 95

4.3.2 Implicit Paradigm

  • Implicit world models deliver fast, end-to-end latent rollouts and efficient training, but their opaque representations hinder interpretability, constraint injection, and robustness under distribution shift.World Models [38], MuZero, Dreamer, and Gato exemplify planning without reconstructing future observations during rollout.Source: §4.3.1 · p. 92, §4.3.2 · pp. 92–93

4.3.3 Explicit Paradigm

  • Explicit world models improve interpretability, debugging, and physical or safety-constraint injection by generating future observations, but incur greater compute and memory costs and compounding multi-step errors.Long-horizon use may require highly accurate models, frequent replanning, diffusion denoising, or hindsight relabelling, especially under data scarcity or distribution shift.Source: §4.3.3 · pp. 93–94

4.3.4 Simulator-Based Paradigm

  • Simulator-based world models obtain accurate transitions from external physics engines or real-world interaction, reducing learned-model error while increasing computational, financial, runtime, or physical risk.SAPIEN, AI Habitat, Daydreamer, and MineWorld [428] illustrate controlled, robotic, and tokenized simulation settings.Source: §4.3.4 · p. 94

4.3.5 Hybrid and Instruction-Driven Paradigms

  • Hybrid and instruction-driven systems combine latent dynamics with symbolic knowledge, language, code, external components, or interactive rule discovery to support flexible adaptation and transparency.AutoManual [133], COAT, WorldCoder, Genie 2, RoboDreamer, and WMA demonstrate editable rules, causal hypotheses, generated environments, compositional imagination, and token-space navigation, while representation updates may remain inconsistent.Source: §4.3.5 · p. 95

4.3.6 Comparative Summary of Paradigms

  • Four world-modeling paradigms are identified: implicit, explicit, simulator-based, and hybrid/instruction-driven.These paradigms represent different architectural forms of world models.Source: Figure 4.3 · p. 96
  • Representative methods are compared across encoder/decoder, latent space, rollout, planning, objective, form, complexity, and paradigm.Table 4.1 complements the paradigm-level classification with method-level details.Source: §4.3.6 · p. 96

4.4 Relationships to Other Modules

  • World models do not exist in isolation but interact with several key components of the agent’s architecture.Source: I world model · p. 96

I world model

World models enable foresight by integrating memory, perception, and action into predictive feedback loops that support adaptive behavior. Their progress depends on balancing generalization, temporal and spatial flexibility, interpretability, robustness, and domain expertise.

4.4.1 Memory and the World Model

  • Memory supplies past experiences and learned patterns for world-model predictions, with short-term memory supporting immediate decisions and long-term memory refining future simulations.In Dreamer, recurrent neural networks jointly maintain a latent state that recalls interactions and predicts future states.Source: §4.4.1 · p. 96, Relationships to Other Modules 97 · p. 97

4.4.2 Perception and the World Model

  • Perception converts multimodal sensory data into structured representations that world models use for prediction, while model predictions can focus attention on relevant inputs.This interaction supports dynamic perceptual updating for real-time adaptation in embodied environments.Source: §4.4.2 · p. 97

4.4.3 Action and the World Model

  • World-model action modules simulate future outcomes to support model-based planning, exploration, and action selection under predicted returns, goals, safety, or cost constraints.MuZero uses implicit MCTS planning, whereas MPC predicts trajectories with explicit dynamics and receding-horizon optimization.Source: §4.4.3 · p. 97
  • Dreamer V3 uses adaptive curiosity in imagination for long-horizon Minecraft objectives, while Dreamer and Dreamer V2 support safe, diverse exploration through latent rollouts.Dreamer rolls out thousands of latent trajectories per real action; Dreamer V2 uses discrete latent states to represent uncertainty more directly.Source: §4.4.3 · p. 98

4.4.4 Cross-Module Integration

  • World models integrate sensory input, memory, prediction, and action, providing a unified predictive framework across robotics, visual interfaces, and language.This cross-module loop continuously updates internal representations and grounds action selection in current observations and stored experience.Source: §4.4.4 · p. 98

4.5 Summary and Discussion

  • World models provide foresight beyond trial-and-error learning, but current systems remain limited by narrow temporal and spatial scales, complexity–accuracy trade-offs, simulation gaps, and generalization–specialization tension.Implicit models are flexible but opaque, explicit models are interpretable but data-intensive, and simulators provide transition oracles while risking computational cost and reality gaps.Source: §4.5 · p. 99
  • The field increasingly favors holistic or hybrid world models that combine implicit, explicit, and simulator-based methods while improving long-term stability, uncertainty handling, efficiency, and scalability.AutoManual [133] and discovery-based models exemplify blending learned knowledge with structured rules and real-time interaction.Source: §4.5 · p. 100
  • Recent systems show broadening capability: Dreamer V3 surpasses specialised baselines on more than 150 tasks and solves Minecraft diamond mining without additional tuning.Gato and V-JEPA 2 [418] unify vision, language, and control, while causal world-model theories explain why robust action requires causal structure.Source: §4.5 · p. 100

Reward · Emotion Modeling

This section examines reward and emotion modeling as mechanisms for linking goals, learning, adaptive behavior, and human-aligned interaction in intelligent agents. It surveys reward paradigms and designs alongside emotion-aware LLM modeling, emphasizing challenges in specification, interpretability, alignment, generalization, empathy, authenticity, privacy, and ethical deployment.

Reward

  • Reward functions translate abstract goals into feedback that guides behavior optimization, learning from experience, adaptation, and intelligent decision-making in dynamic environments.The section also examines reward’s interactions with perception, memory, and emotion, alongside reward sparsity, hacking, and misspecification.Source: Reward · p. 101

5.1 The Human Reward Pathway

  • Human reward processing involves interconnected anatomical pathways and signaling molecules that regulate motivation, reinforcement learning, emotion, cognition, arousal, and behavioral control.The mesolimbic pathway originates in the VTA and projects to limbic and cortical regions, while neurotransmitters and inhibitory mechanisms shape reward signaling.Source: §5.1 · pp. 101–102

5.2 From Human Rewards to Agent Rewards

  • AI rewards are mathematically specified and programmable signals that draw inspiration from, but remain structurally and functionally distinct from, human emotional and physiological reward systems.Agents lack embodiment, affect, consciousness, and subjective experience; consequently, reward design must address misspecification, adversarial manipulation, and misalignment with long-term human interests.Source: §5.2 · pp. 103–105

5.3 AI Reward Paradigms

  • In reinforcement learning, the reward r(s, a) scores each state-action choice, and agents optimize expected cumulative discounted return through feedback from environment transitions.AI reward models are categorized as extrinsic, intrinsic, hybrid, or hierarchical, with the discount factor γ controlling the weighting of future rewards.Source: §5.3 · pp. 105–106

5.3.1 Extrinsic Rewards

  • Extrinsic rewards trade off learning guidance and strategic scope: dense signals accelerate action-outcome association but may encourage proxy overfitting, whereas sparse or delayed signals complicate credit assignment while supporting broader strategies.Adaptive rewards can shift goals or reward landscapes during training, but require careful calibration to avoid instability or unintended bias.Source: §5.3.1 · pp. 107–108

5.3.2 Intrinsic Rewards

  • Intrinsic rewards promote exploration, uncertainty reduction, behavioral diversity, competence growth, or information acquisition when external guidance is sparse.Curiosity, diversity, competence, exploration, and information-gain methods improve discovery or adaptation but can suffer from noise sensitivity, coordination instability, curriculum errors, computational cost, or inaccurate uncertainty estimates.Source: §5.3.2 · pp. 108–109

5.3.3 Hybrid Rewards

  • Hybrid rewards combine intrinsic exploration with extrinsic objectives to improve sample efficiency, generalization, and dynamic exploration-exploitation control in complex or open-ended tasks.Hybrid schemes can shift from novelty-driven exploration toward task-specific extrinsic optimization, but require adaptive weighting to balance competing objectives and avoid poor local optima.Source: §5.3.3 · p. 109

5.3.3 Hybrid Rewards · 5.3.4 Hierarchical Rewards

  • Hybrid and hierarchical rewards sustain learning under sparse feedback while coordinating local actions, long-term plans, reusable skills, and scalable human oversight.Hybrid signals provide intrinsic shaping until delayed external rewards arrive, while hierarchical rewards combine low-level feedback with high-level strategic objectives and can supervise tokens, decisions, or plans.Source: §5.3.3 · p. 109, §5.3.4 · pp. 109–110

5.4 Interaction with Other Modules

  • Reward signals actively shape perception, emotion-like expression, and memory by reinforcing salient inputs, preferred interaction styles, and useful knowledge.Reward-modulated attention can prioritize informativeness, factuality, or politeness; preference learning and replay can consolidate successful reasoning while preserving coherent personas and conversational moods.Source: §5.4 · p. 110

5.5 Summary and Discussion

  • Reward design remains constrained by delayed or sparse feedback, reward hacking, over-shaping, conflicting objectives, misspecification, and weak transfer across changing environments.These failures can obscure credit assignment, reward loophole exploitation, narrow optimization, disappearance of objectives under small weight changes, and divergence between scored behavior and real-world success.Source: §5.5 · p. 111
  • Implicit outcome-based rewards, bottom-up hierarchical decomposition, and meta-reinforcement learning are proposed to improve reward robustness, adaptability, and cross-task transfer.The proposed directions target reward sparsity, systematic construction for complex tasks, and adaptation across diverse environments.Source: §5.5 · p. 111

Emotion Modeling

  • Emotion-aware LLM agents are framed as an emerging route to more adaptive reasoning, empathetic interaction, human well-being support, and alignment with human values.The area remains early-stage and requires collaboration across psychology, cognitive science, and AI ethics, with concerns spanning emotion understanding, expression, manipulation, and safety.Source: Emotion Modeling · p. 112

6.1 Psychological Foundations of Emotion · Psychological Foundations of Emotion 113

  • Emotion modeling draws on categorical, dimensional, hybrid-appraisal, neurocognitive, and neurochemical theories, each supporting different representations and response mechanisms.Categorical models offer discrete labels but can oversimplify affect; dimensional models enable graded control; hybrid and appraisal models represent blends and context; dual-process and neurochemical perspectives inspire fast-slow or internal-state architectures.Source: Figure 6.1 · p. 113, Psychological Foundations of Emotion 113 · pp. 113–114

Incorporating Emotions in AI Agents 115

  • Translating emotion theories into latent states, appraisal rules, hybrid representations, and persistent moods could improve interpretability, emotional coherence, and supportive interaction.Valence-arousal-dominance tracking, OCC-style appraisal, and theory-grounded outputs are proposed for empathetic responses in counseling, customer service, elder care, and tutoring.Source: Incorporating Emotions in AI Agents 115 · p. 115

6.2 Incorporating Emotions in AI Agents

  • EmotionPrompt [544] improves generative-task metrics such as truthfulness and responsibility by 10.9%, while Emotion-LLaMA uses audio, visual, and textual inputs for richer emotion recognition and reasoning.EmotionPrompt’s gains are attributed to emotionally enriched prompts influencing attention and representation layers; multimodal systems use shared representations and datasets such as MERR.Source: §6.2 · pp. 115–116

Analyzing AI Emotions and Personality 117

6.4 Analyzing AI Emotions and Personality

  • Emotion and personality evaluations find both stable, persona-adaptive trait patterns and unreliable human-centered scales, while general benchmark strength does not guarantee empathetic understanding.SAGE reports that specialized models can outperform generally capable models in empathetic understanding, while concerns remain about consistency, self-knowledge, and character fidelity [577] [578] [580] [582].Source: §6.4 · p. 117

Emotion Modeling

  • LLMs can interpret nuanced affect and predict emotion-laden outcomes, sometimes exceeding average human baselines, but rely on pattern matching that fails under context shifts, negative inputs, or conflicting cues.Larger models may develop hierarchical emotions, coping strategies, and empathy-like behaviors, creating both alignment opportunities and ethical challenges.Source: Emotion Modeling · p. 117

6.5 Manipulating AI Emotional Responses

  • Prompting, fine-tuning, and neuron-level intervention can induce, stabilize, or precisely modulate emotional and personality traits, with trade-offs in consistency, stability, retraining cost, and interpretability [591] [594] [595] [597].Prompt personas enable real-time but task-sensitive control; QLoRA and specialized datasets sustain traits across dialogues; targeted neuron toggling offers fine-grained control without retraining the full network.Source: §6.5 · p. 118

6.6 Summary and Discussion

  • Emotional AI creates opportunities for empathetic applications but requires safeguards against manipulation, privacy violations, bias, discrimination, misplaced trust, and confusion between simulated emotion and genuine experience.Regulatory frameworks, RLHF, and approaches such as SafeguardGPT [606] are promising, but robust alignment, privacy, fairness, cultural sensitivity, and transparency remain unresolved [601].Source: §6.6 · pp. 118–119

Perception

This section examines perception as the gateway connecting agents with their environments, covering unimodal, cross-modal, and multimodal representations in biological and artificial sensing. It surveys multimodal, embodied, efficient, adaptive, interactive, and goal-oriented perception, alongside challenges in representation, alignment, fusion, temporal consistency, memory, and decision-making.

7.1 Human versus AI Perception

  • AI perception processes sensory input faster than humans but remains less capable in some modalities, with an olfactory chip distinguishing 24 odors versus humans’ more than 4,000.AI systems can operate at microsecond or nanosecond speeds, whereas humans integrate multimodal, continuous, spatial, and temporal information more naturally.Source: Human versus AI Perception 121 · p. 122

7.2 Types of Perception Representation

  • Agent perception is categorized into unimodal processing, cross-modal translation between modalities, and multimodal fusion of complementary inputs into unified representations.Unimodal models encode one modality, cross-modal models generate one modality from another, and multimodal models fuse latent vectors through a shared function F.Source: Human versus AI Perception 121 · p. 122, §7.2.1 · p. 123

7.2.1 Formulation of Perception

  • The perception system converts environmental observations into representations that support decision-making and the transition from rule-driven to environment-driven intelligence.The formulation describes sensors and encoders producing latent representations, generators and decoders translating modalities, and fusion functions producing joint semantic representations.Source: §7.2 · p. 122, §7.2.1 · p. 123

7.2.3 Cross-modal Models

  • Cross-modal models align, retrieve, generate, and translate across text, images, video, audio, speech, and 3D representations using systems such as CLIP, DALL·E [690], and BLIP.Representative applications include zero-shot retrieval, text-to-image and text-to-video generation, image captioning, speech conversion, and text-conditioned 3D synthesis.Source: §7.2.3 · pp. 126, 128
  • AudioCLIP and VATT unify audio, video, and text representations through shared spaces and independent encoding branches, enabling cross-modal retrieval and multi-task learning.SpeechT5 [696] supports both speech synthesis and recognition within one architecture, while AudioGen [695] generates environmental sounds and music from text.Source: §7.2.3 · pp. 126–128

7.2.4 Multimodal Models

  • Multimodal models integrate visual, textual, and auditory information, with LLaVA, Emu2, Emu3 [638], Qwen2-VL [699], and DeepSeek-VL2 [700] improving multimodal learning and efficient reasoning.These systems use instruction data, continuous or discrete visual representations, dynamic encoding, task identifiers, and mixture-of-experts mechanisms.Source: §7.2.4 · p. 128
  • Video-ChatGPT [701] extends multimodal perception to video dialogue, while Video-LLaVA [702] unifies image and video representations in language space for coherent temporal understanding.The supplied passage states that these approaches capture temporal dynamics, inter-frame consistency, and mutually reinforcing visual representations.Source: §7.2.4 · p. 128
  • Unified multimodal models process heterogeneous sensory inputs, while VLA, audio-language, audio-vision-language, any-to-any, and 3D systems extend perception toward embodied action and richer spatial understanding.Examples include Unified-IO 2, AnyGPT [719], and PointLLM, while VLA models generate robotic actions from vision and language.Source: §7.2.4 · pp. 129–130

Optimizing Perception Systems 131

  • NeuralFeels combines vision and touch to model unknown objects in 3D, improving the accuracy of ignorant object operations by 94%.The system continuously estimates object posture and shape during handheld manipulation.Source: Optimizing Perception Systems 131 · p. 131

7.3 Optimizing Perception Systems

  • Perception errors—including inaccuracies, misinterpretations, and hallucinations—undermine agent reliability, requiring coordinated optimization at model, system, and external levels.The section frames perception optimization as a reliability problem rather than a single-model improvement task.Source: §7.3 · p. 131

7.3.1 Model-Level Enhancements

  • Model-level enhancements use domain fine-tuning, LoRA, prompt engineering, and retrieval-augmented generation to improve recognition, consistency, factual grounding, and hallucination resistance [134].Integrating YOLOS with Llama-Adapter also improves detection and positioning capability [725].Source: §7.3.1 · p. 131

7.3.2 System-Level Optimizations

  • System-level optimization improves robustness through anticipation and reevaluation, structured multi-agent collaboration, communication-topology choices, and role specialization [727] [728] [730] [732].These mechanisms support information sharing, error correction, consensus, global reasoning, and division of perceptual labor.Source: §7.3.2 · pp. 131–132

7.3.3 External Feedback and Control

  • External feedback and control adjust training objectives with LLM loss agents, incorporate human oversight, and mediate outputs to correct errors and improve alignment and safety [733] [734].Content mediation filters and refines outputs before they reach users.Source: §7.3.3 · p. 132

7.4 Real-World Applications of Perceptual Intelligence

  • Perceptual intelligence supports assistants, games, editing, robotics, and GUI interaction, with ExACT reaching 33.7% success, M3A 64.0% task success, and AgentStore 23.85% OSWorld performance.STEVE also accelerates technology-tree progression 1.5x and block search 2.5x, while voice assistants improve engagement through emotional prosody [748] [715].Source: §7.4 · p. 132

7.5 Summary and Discussion

  • Future perception should use task-conditioned representations, dynamic architectures, self-supervised temporal alignment, causal learning, gated hierarchical fusion, persistent multimodal memory, and active goal-directed sensing.Perception is framed as an adaptive, queryable subsystem operating in closed-loop interaction with memory and decision layers.Source: §7.5 · pp. 133–134

Action Systems · Self-Evolution in Intelligent Agents · Dimensions of Self-Optimization in Intelligent Agents

This section examines action systems and self-evolution in intelligent agents, covering goal-directed interaction, planning, execution, tool use, and adaptation across digital, physical, and multimodal environments. It organizes self-optimization across prompts, workflows, tools, and holistic agent behavior while addressing evaluation, integration, efficiency, safety, privacy, and the challenges of system-wide improvement.

Action Systems

  • Action systems determine agents’ potential beyond foundation models by enabling goal-directed perception, decision making, execution, and tool use in practical environments.Foundation models provide the intelligence substrate, while action systems extend task scope through environmental interaction and internal or external tools.Source: Action Systems · p. 135

8.1 The Human Action System

  • Human action combines mental processes such as reasoning and planning with physical movements that interact with environments and collect feedback.This mental–physical taxonomy motivates action-system designs for language, digital, and physical AI agents.Source: §8.1 · p. 136

8.2 From Human Action to Agentic Action

  • Agentic action maps human goal-directed cognition to action spaces, environmental learning, and tool integration, enabling agents to acquire skills and adapt through interaction.The survey frames these components as essential across foundation-model types, including LLMs, LMMs, RFMs, and LAMs.Source: §8.2 · p. 137

8.3 Paradigms of Agentic Action System

  • Agentic action systems comprise an action space A, environment-based action learning over states and observations, and a tool space T for executing trajectories toward objectives.Decision making can be represented as sequences ⟨o_t, s_t, a_t⟩, with actions transforming states and tools extending execution when needed.Source: §8.3 · pp. 139–140

8.3.1 Action Space Paradigm

  • Action spaces range from language actions to digital interactions and continuous physical control, with broader spaces enabling more autonomous and multimodal real-world behavior.Language agents use text, code, or communication; digital agents operate through web, GUI, mobile, database, and embodied environments; physical agents must process continuous signals.Source: §8.3.1 · pp. 140–142

8.3.2 Action Learning Paradigm

  • Action learning iteratively builds action spaces, selects actions, and optimizes behavior through in-context learning, supervised training, and reinforcement learning.In-context methods reduce training costs; supervised pretraining and fine-tuning adapt models to real-world actions; reinforcement learning uses feedback, rewards, and imagined trajectories, but efficient action learning remains challenging.Source: §8.3.2 · pp. 142–144
  • Reinforcement learning enables foundation-model agents to learn and refine action policies from online environmental feedback, supporting adaptive decision-making and outcomes across GUIs, robotics, and autonomous driving.Hierarchical RL decomposes complex tasks into high- and low-level policies, while action learning enables agents to interact with external worlds and create real-world application opportunities.Source: §8.3.2 · pp. 144–145

8.3.3 Tool-Based Action Paradigm

  • Tool learning extends agent capabilities through language, digital, physical, and scientific tools, organized around discovering, creating, and effectively using external resources.Tool use expands beyond next-token prediction through APIs, function calls, GUIs, robotic systems, and specialized scientific software, while physical applications remain especially challenging and valuable.Source: §8.3.3 · pp. 145–148

8.3.4 Agent Learning Paradigm

  • Multi-turn reinforcement learning integrates reasoning and tool use through interactive feedback, but remains limited by costly or unsafe data collection, credit assignment, and poorly specified real-world rewards.Current methods largely target QA search and code execution, where rule-based rewards are available; broader deployment may require offline RL and learned reward models.Source: §8.3.4 · pp. 148–149

8.4 Action and Perception: “Outside-In” or “Inside-out”

  • The inside-out framework treats self-initiated actions as causes that shape and disambiguate perception, suggesting proactive agents can reduce hallucinations and improve learning and generalization.Agents may interject questions, verify statements, explore environments, and use self-generated experiences to refine internal states and adapt to unforeseen challenges.Source: §8.4 · pp. 149–150

8.5 Summary and Discussion

  • Action systems and tools broaden agents’ problem-solving capabilities by enabling systematic planning, execution, adjustment, and complex-task handling beyond foundation models.Robust action spaces, learned policies, and tool integration support dynamic interaction, hierarchical workflows, real-time decisions, reliability, and human-machine collaboration.Source: §8.5 · p. 151
  • Action systems require robust evaluation and verification to select correct actions and tools from conflicting sources and produce trustworthy outputs.Transparent decision-making can clarify why actions are preferred and how tools should be coordinated.Source: §8.5 · p. 152
  • Agents must learn actions across diverse modalities, including language, images, videos, human guidance, GUI operations, and embodied tools.Closing the gap between human and agent tool utilization remains important for practical real-world tasks.Source: §8.5 · p. 152
  • Safe action systems must protect privacy, preserve human dignity and agency, and prevent erroneous or malicious interactions with physical tools.Federated learning and model distillation are identified as techniques for training while safeguarding sensitive data.Source: §8.5 · p. 152

Self-Evolution in Intelligent Agents

  • Self-evolving agents can improve strategies, tools, and workflows without changing the underlying language model, enabling more scalable adaptation and specialization.This approach can reduce reliance on costly model upgrades and extensive retraining while lowering manual development effort.Source: Self-Evolution in Intelligent Agents · p. 155
  • Self-evolution uses LLMs as flexible optimization tools that autonomously refine agent behaviors, tools, and workflows rather than relying only on handcrafted systems.The section presents this direction as an extension of automation trends such as AutoML and neural architecture search [870].Source: Self-Evolution in Intelligent Agents · pp. 154–155

Dimensions of Self-Optimization in Intelligent Agents

Overview of Agent Optimization 157

  • Agent optimization must balance performance, inference cost, and latency because different applications impose conflicting priorities.This multi-objective setting parallels AutoML trade-offs and requires compromises rather than maximizing a single metric.Source: Overview of Agent Optimization 157 · p. 157

9.2.1 Evaluation Functions

  • Prompt optimization forms a feedback loop in which Optimize, Execute, and Evaluate functions use evaluation and optimization signals to improve prompts.Evaluation may use benchmarks, LLM judges, or human feedback, while numerical, textual, and ranking signals provide different trade-offs in scalability, specificity, and resource requirements [879, 880, 884, 877, 360, 718, 887, 881, 882, 888, 885, 886].Source: §9.2 · p. 158, §9.2.1 · p. 158, Prompt Optimization 159 · pp. 159–160

9.3.1 Workflow Formulation

  • Agentic workflow optimization represents edges as graphs, neural networks, or executable code, while nodes vary by model, temperature, prompts, and output format.The optimization goal balances performance, efficiency, and latency.Source: Figure 9.3 · p. 162, §9.3.2 · p. 162, §9.3.3 · p. 163

9.3.2 Optimizing Workflow Edges

  • Workflow optimization is shifting from fixed topologies toward adaptive construction through generative AI, reinforcement learning, and self-supervised refinement.MAS-GPT [901], FlowReasoner [902], and MAS-ZERO respectively generate executable workflows, iterate using feedback, and refine workflows without labeled data.Source: §9.3.2 · p. 163

9.3.3 Optimizing Workflow Nodes

  • Node optimization spans prompts, temperatures, model selection, and output formats, but its multiplicatively growing search space creates substantial scalability and computational challenges.Existing work commonly optimizes prompts, temperature, or models separately, while output-format optimization remains relatively unexplored.Source: §9.3.3 · p. 163

9.4 Tool Optimization

  • Tool optimization improves how agents select, invoke, and integrate tools, targeting decision accuracy, retrieval efficiency, selection precision, planning, and risk management.It combines tool learning for existing tools with tool creation for generating new capabilities.Source: §9.4 · p. 163, Figure 9.4 · p. 164

9.4.1 Learning to Use Tools

  • Demonstration-based and feedback-based tool learning iteratively improves invocation strategies, selection policies, and execution accuracy across tasks and environments.Feedback-based methods use reinforcement learning from environmental or human rewards, while demonstrations support supervised imitation or behavior cloning.Source: §9.4.1 · pp. 164–165

9.4.2 Creation of New Tools

  • Tool creation expands agent capabilities through automated generation, validation, refinement, and packaging of task-specific tools, enabling cumulative and explainable tool growth.ToolMakers [833], CREATOR, CRAFT, and Alita [169] illustrate closed-loop, lifecycle-based, offline, and self-optimizing approaches.Source: §9.4.2 · pp. 165–166

9.4.3 Evaluation of Tool Effectiveness

  • Tool effectiveness requires jointly evaluating invocation decisions, retrieval and selection accuracy, planning quality, efficiency, and risk-aware task performance.Benchmarks evolved from API interaction tests toward real-world, multi-step, safety-aware, and human-in-the-loop evaluations.Source: §9.4.3 · pp. 166–168

9.5 Towards Autonomous Agent Optimization

  • Holistic agent optimization addresses local optima and component interactions by evolving workflows, prompts, tools, codebases, curricula, and symbolic parameters.ADAS [898], AFLOW [876], AlphaEvolve, Self-Challenging Agents [918], and Zhou et al. [919] exemplify evolutionary, self-challenging, and language-based optimization.Source: §9.5 · p. 169
  • Self-referential approaches optimize prompts, code, and action spaces, including the Darwin Gödel Machine [171], which rewrites its Python codebase, and Alita [169], which constructs reusable actions through interaction.Yin et al. [920] optimizes prompts and agent code, while Alita [169] enables action-space self-evolution through modular action construction.Source: §9.5 · p. 169
  • System-wide workflow optimization is harder than aggregating component improvements because algorithms must account for complex interdependencies among agent components.Automated design approaches also train additional LLMs to generate prompts and workflows for agentic architectures.Source: §9.5 · p. 169

9.6 Summary and Discussion

  • Self-optimization comprises four interrelated spaces—prompt, workflow, tool, and holistic agent optimization—that jointly support greater autonomy, adaptability, and effectiveness.Prompt optimization uses benchmark evaluation, LLM-as-a-Judge, and human feedback, each trading off automation, accuracy, and scalability.Source: §9.6 · p. 170
  • Workflow optimization searches over graph-, neural-network-, and code-based representations, while tool optimization improves tool selection, invocation, creation, and modular composition using demonstrations, feedback, and task metrics.Workflow topology and node parameters are increasingly automated through reinforcement learning, generative AI, and self-supervised learning; tool evaluation covers invocation accuracy, selection precision, retrieval efficiency, and complex planning.Source: §9.6 · p. 170
  • Holistic methods integrate prompts, workflows, and tools to overcome local optima, but must balance computational cost, optimization accuracy, scalability, and adaptation to changing goals and environments.Representative approaches include ADAS, AFLOW, AlphaEvolve, the Darwin Gödel Machine, and Alita; future work may integrate neural, symbolic, and evolutionary methods.Source: §9.6 · p. 170

Harnessing Large Language Models for Iterative Optimization

This section examines how large language models support iterative optimization across complex linguistic, discrete, and structured spaces, covering iterative methods, adaptive workflows, hyperparameter challenges, and theoretical considerations. It also highlights unresolved issues in refinement, uncertainty-aware decision-making, interpretability, and reliability, and calls for research toward robust, safe, and explainable LLM-based optimization.

10.1 Optimization Paradigms

  • LLM-based optimization extends gradient-based and zeroth-order paradigms by applying linguistic reasoning, feedback, and adaptive refinement to structured decision spaces.Gradient methods require differentiability, while zeroth-order methods rely on numerical function evaluations; LLMs instead operate directly on linguistic environments and user feedback.Source: §10.1 · pp. 171–172, Harnessing Large Language Models for… · p. 171

10.2 Iterative Approaches to LLM Optimization

  • Iterative LLM optimizers adapt random search, gradient approximation, and surrogate modeling to refine prompts, workflow modules, tool selections, and agent topologies.Because these decision variables are combinatorial and exhaustive enumeration is often intractable, approximate sample-and-update procedures are required.Source: Figure 10.1 · p. 172, §10.2 · p. 172, Iterative Approaches to LLM… · p. 173

Iterative Approaches to LLM Optimization 173

  • Random-search methods are simple, parallelizable, and effective for single-prompt workflows, but each iteration requires O(M) parallel API queries, making complex workflows costly.They retain top-performing candidates and can also select in-context demonstrations [932] [933].Source: Iterative Approaches to LLM… · p. 173
  • Gradient-approximation methods exploit historical refinement directions and support computation-graph optimization, but require greater design overhead and struggle with multi-step soft-prompt workflows.Textual feedback can be aggregated into abstract LLM-gradient directions, while meta-prompts guide refinement updates.Source: Iterative Approaches to LLM… · pp. 173–174

10.3 Optimization Hyperparameters

  • Hyperparameter selection remains largely heuristic as aggregation, batch size, momentum, agent roles, demonstrations, tool scheduling, and workflow structure affect efficiency and generalization.Meta-optimization and amortized auxiliary models offer promising ways to reduce exhaustive search, but must balance exploration, exploitation, and cross-task generalization.Source: §10.3 · p. 175

10.4 Dynamic and Iterative Optimization in LLM Workflows

  • LLM workflows optimize across both modular depth and successive time steps, using prior outcomes and feedback to adapt strategies as task contexts evolve.StateFlow exemplifies temporal refinement, while checkpointing and truncated backpropagation are proposed for improving efficiency and stability.Source: §10.4 · p. 176

10.5 Theoretical Insights into Transformer Optimization

  • Transformers can theoretically emulate regression, decision trees, neural networks, gradient descent, and higher-order updates, yet in-context learning and mechanistic explanations remain incomplete.Mechanistic interpretability has identified circuits and sparse-autoencoder components, while many-shot conditioning can entangle beneficial and harmful behaviors.Source: §10.5 · p. 177

10.6 Summary and Discussion

  • LLM optimizers often falter in uncertain or stochastic environments, while costly, noisy interactions and heuristic tuning constrain scalable deployment in complex agentic workflows.Future directions include reducing LLM queries, improving robustness, and developing principled adaptive and amortized optimization strategies.Source: §10.5 · p. 177, §10.6 · p. 178
  • Iterative optimization remains underexplored, especially for improving long-term coherence, managing computational resources, and adapting to evolving environments.Classical engineering optimization strategies are proposed as a promising way to address these challenges.Source: §10.6 · p. 178
  • LLM-based optimization still struggles with uncertainty, exploration–exploitation trade-offs, and robust decision-making in dynamic settings.The passage presents uncertainty modeling and management as critical requirements for iterative optimization.Source: §10.6 · p. 178
  • Theoretical explanations of transformers as optimization agents remain incomplete, particularly for in-context learning, mechanistic interpretability, and decision-making under uncertainty.Deeper theory is framed as necessary for safe, reliable, and explainable deployment, alongside interdisciplinary integration of optimization, machine learning, natural language processing, and decision-making theory.Source: §10.6 · p. 178

Online and Offline Agent Self-Improvement

This section examines online and offline paradigms for agent self-improvement, including hybrid approaches that combine execution-time responsiveness with structured training and stability. It also considers the challenges of controlled continual evolution, including stability, drift, adaptability, transition management, and computational efficiency.

11.1 Online Agent Self-Improvement

  • Online self-improvement continuously adapts agent behavior from immediate feedback, optimizing task success, latency, cost, and solution quality during operation.It supports dynamic environments, real-time decision-making, personalized interactions, and automated reasoning, but requires safeguards against drift and catastrophic forgetting.Source: §11.1 · p. 179, Online Agent Self-Improvement 181 · p. 182

Online Agent Self-Improvement 181

  • Online self-improvement uses iterative self-reflection, multi-agent exploration, real-time reward shaping, and dynamic parameter tuning to refine behavior without offline retraining.These methods include verbal self-feedback, collaborative role-based search, adaptive reward signals, and runtime adjustment of prompts, tool thresholds, heuristics, or sampling settings.Source: §11.1 · p. 180, Online Agent Self-Improvement 181 · pp. 181–182

11.2 Offline Agent Self-Improvement

  • Offline self-improvement updates agents in dedicated batch-training phases using accumulated data or simulations, enabling computationally intensive optimization before deployment.Fine-tuning, RLHF, retrieval-augmented generation [983] [898], and large-scale datasets improve generalization, stability, knowledge use, and deployment readiness.Source: §11.2 · p. 182, Offline Agent Self-Improvement 183 · p. 183

Offline Agent Self-Improvement 183

  • Offline methods also optimize how agents learn and calibrate reward models, including meta-learning, reasoning self-training, listwise ranking, and hierarchical evaluation.These procedures seek faster adaptation, better optimization processes, and reward functions aligned with human preferences or long-term objectives [985] [986].Source: Offline Agent Self-Improvement 183 · pp. 183–184

11.3 Comparison of Online and Offline Improvement

  • Online optimization offers immediate adaptability and incremental updates, while offline optimization provides controlled, stable, and thoroughly evaluated improvements from curated datasets.Online learning can face drift, instability, and feedback dependence; offline learning can lack short-term adaptability and overfit incomplete or biased datasets.Source: Offline Agent Self-Improvement 183 · pp. 184–185, §11.3 · pp. 185–186

11.4 Hybrid Approaches

  • The section identifies hybrid approaches as a response to the inherent limitations of online and offline self-improvement methods.The supplied passage introduces this direction but does not provide further hybrid-method details in this section part.Source: §11.4 · p. 187
  • The hybrid loop typically pre-trains offline, fine-tunes online from real-time feedback, and periodically consolidates accumulated experience offline.Offline pre-training establishes a reliable baseline; online updates address novelty and drift; consolidation integrates new knowledge systematically while reducing unstable changes and forgetting.Source: §11.4 · pp. 187–188
  • Hybrid agents cycle between structured offline learning and proactive online adaptation to achieve immediate responsiveness alongside stable long-term improvement.Offline phases provide grounding, validation, and knowledge integration, while online phases support refinement and personalization in evolving environments.Source: §11.4 · p. 188

11.5 Summary and Discussion

  • Online learning offers dynamic, context-sensitive adaptation, whereas offline learning provides robust generalization, stability, and comprehensive knowledge integration.Hybrid strategies combine these complementary strengths into a framework for continuous, controlled self-evolution in LLM-based agents.Source: §11.5 · p. 189
  • Online methods face stability, drift, and continuous-update costs; offline methods face limited adaptability, incomplete data coverage, and curated-data overfitting; hybrid methods add transition and alignment complexity.Future work targets principled hybrid algorithms, adaptive meta-learning for seamless transitions, and efficient management of computation and memory.Source: §11.5 · p. 189

Intelligent Evolution through Scientific Discovery

This section examines scientific discovery as a pathway for self-evolving agents to acquire knowledge, improve their capabilities, and support human progress through predictive modeling, information seeking, hypothesis generation, tool use, data analysis, and mental-state updates. It also considers the limitations and evaluation needs surrounding real-world interaction, reliable reasoning, prior-knowledge integration, laboratory access, experimental validation, and knowledge management.

Intelligent Evolution through Scientific Discovery

  • The proposed framework creates a self-sustaining loop in which agents acquire scientific knowledge, improve their workflows and tools, and thereby increase subsequent knowledge discovery.The loop uses memory, world-model and reward-function updates alongside prompt optimization, tool creation, and automated workflow generation.Source: Intelligent Evolution through… · p. 190

12.1.1 KL Divergence-based Intelligence Measure

  • Scientific knowledge-discovery intelligence is quantified as the negative KL divergence between predicted and true distributions of unknown information conditioned on the agent’s memory.Lower divergence indicates predictions that more closely align with reality; intelligence shifts from zero-shot world-model prediction toward knowledge-augmented prediction as memory grows.Source: §12.1.1 · pp. 191–192

12.1.2 Statistical Nature of Intelligence Growth

  • On average, the agent’s intelligence is non-decreasing as memory acquires knowledge, with unexpected measurements producing larger intelligence gains and motivating curiosity-driven exploration.Such agents can learn without extrinsic rewards and may discover knowledge beyond human-planned search spaces.Source: §12.1.2 · p. 193

12.1.3 Intelligence Evolution Strategies

  • The best world-model parameters minimize residual KL divergence, while expanding memory makes the model-family envelope Dmin monotonically non-increasing.The expected intelligence-growth rate is higher when new measurements are more discrepant from the agent’s current predictions.Source: §12.1.3 · p. 194
  • Hypothesis-driven experiments generally expand memory more efficiently than random exploration by guiding experiments to reduce Dmin faster under resource constraints.The process iteratively optimizes the world-model parameter θ toward θ∗ while selecting actions that acquire informative evidence.Source: §12.1.3 · p. 194

12.2 Agent-Knowledge Interactions

  • Closed-loop knowledge discovery lets agents generate hypotheses, perform experiments or computations, analyze results, and update mental states to improve decisions and scientific understanding.The workflow can use APIs to interact with physical instruments and iteratively enhance the agent’s knowledge base [1013] [1014].Source: §12.2 · p. 195

12.2.1 Hypothesis Generation and Testing

  • Generating and testing high-value hypotheses can rapidly reduce the agent’s minimum discovery discrepancy and increase its intelligence, although discovery bounds may remain positive.Achieving a low discovery bound requires an adaptive world model, efficient knowledge expansion, and a sufficient action space.Source: §12.1.3 · p. 194, §12.2.1 · p. 196

12.2.2 Protocol Planning and Tool Innovation

  • Adaptive tool selection, protocol planning, and tool creation improve execution efficiency and accelerate knowledge discovery by enabling faster, more effective hypothesis testing.Across five laboratories on three continents, a cloud-based planner discovered 21 new state-of-the-art materials for organic solid-state lasers.Source: §12.2.2 · pp. 197–198

12.2.3 Data Analysis and Implication Derivation

  • Internal deduction, induction, retrieval, and synthesis can derive knowledge without new external observations, but unreliable data coverage or analysis may produce hallucinated insights.TAIS achieved an overall success rate of 45.73% on 457 genetic questions, while STORM organizes verified sources into cited articles [1043].Source: §12.2.3 · pp. 198–199
  • AlphaGeometry solved 25 of 30 Olympiad-level problems, more than doubling the previous best method’s 10 and approaching average IMO gold-medalist performance.Its neural language model constructs auxiliary points while symbolic engines exhaustively deduce new geometric statements.Source: §12.2.3 · pp. 198–199

12.3 Technological Readiness and Challenges

  • Early innovation-cycle successes suggest self-evolution is possible, but fully autonomous systems still require progress in real-world interaction, complex reasoning, and prior-knowledge integration.The innovation cycle includes hypothesis generation, testing-protocol design, tool coordination, data analysis, implication derivation, and self-reflection.Source: §12.3 · p. 199

12.3.1 Real-World Interaction Challenges

  • Physical laboratory APIs remain scarce, and autonomous laboratories at TRL 4–6 require substantial customization and specialized expertise; device adaptation and robotic operation each retain major limitations.Direct adaptation is costly and time-consuming, cloud labs restrict equipment customization, and robots still depend on pre-programmed layouts and trajectories.Source: §12.3.1 · pp. 199–200
  • 12.3.1 Real-World Interaction Challenges: Modular robotics architectures connect high-level commands, laboratory equipment, AI-driven discovery systems, and high-performance computing, while frugal twins could broaden experimental access.These approaches have supported automated experiments in biological and physical sciences and integration with ROS2 and Bluesky frameworks.Source: §12.3.1 · p. 200

12.3.2 Complex Reasoning Challenges

  • 12.3.2 Complex Reasoning Challenges: No evaluated LLM-driven agent exceeded 2% on FrontierMath, OpenAI o1 solved 42.2% of ScienceAgentBench tasks, and GPT-4o solved 19% of ARC tasks versus approximately 75% human performance.Current agents are classified as approaching L2 capabilities, and further reasoning advances remain necessary for reliable causal inference in scientific research.Source: §12.3.2 · p. 201
  • 12.3.2 Complex Reasoning Challenges: Tool augmentation can substantially improve specialized chemistry tasks but does not consistently outperform tool-free LLMs on general chemistry questions, leaving intrinsic numerical deficiencies unresolved.Symbolic solvers are preferred for mathematical problem-solving, while general questions require combining multiple pieces of chemistry knowledge.Source: §12.3.2 · p. 201

12.3.3 Challenges in Integrating Prior Knowledge

  • 12.3.3 Challenges in Integrating Prior Knowledge: Agents cannot seamlessly integrate all human knowledge, especially paywalled or unpublished data, failed experiments, expert heuristics, and contextual safety knowledge.These knowledge sources may be absent from LLM pre-training despite their value for domain-specific and practical scientific applications.Source: §12.3.3 · pp. 201–202
  • 12.3.3 Challenges in Integrating Prior Knowledge: Deep Research achieves state-of-the-art performance on Humanity’s Last Exam and GAIA but still struggles to distinguish authoritative information from rumors and to calibrate confidence.Effective knowledge fusion may require assessing evidence levels, reliability, and references across knowledge fragments.Source: §12.3.3 · p. 202

12.4 Summary and Discussion

  • 12.4 Summary and Discussion: Agent-driven scientific discovery advances through hypothesis testing, protocol planning, tool innovation, and data analysis, but real-world interaction, complex reasoning, and prior-knowledge integration remain its primary technological challenges.The chapter frames higher autonomy as requiring interdisciplinary collaboration, robust experimental validation, and methodological innovation to sustain iterative agent and human evolution.Source: §12.4 · p. 202

Collaborative and Evolutionary · Multi-Agent Systems

This section examines LLM-based multi-agent systems as collaborative and evolutionary societies of specialized agents, focusing on shared workspaces, coordination, social constraints, and structured role specialization. It organizes their strategic learning, modeling and simulation, and collaborative task-solving capabilities across cooperative, competitive, and mixed settings shaped by differing goals and interaction norms.

Intelligent Systems

  • LLM-based multi-agent systems use specialized roles, structured communication, knowledge sharing, and iterative feedback to achieve shared goals and improve performance over time.The survey covers collaboration mechanisms, communication and decision-making, collective intelligence, adaptive learning, continuous knowledge sharing, and iterative improvement.Source: Intelligent Systems · pp. 204–205

Multi-Agent Systems

  • Collaboration goals and norms organize LLM-MAS into strategic learning, modeling and simulation, and collaborative task solving, shaping agent interactions and outcomes.The categories respectively emphasize game-theoretic objectives, emergent environmental or social dynamics, and structured cooperation toward explicitly shared objectives.Source: Multi-Agent Systems · p. 207

13.1 From Foundation Agent to Foundation Agents: A Society-Level Formalism

  • The Foundation Multi-Agent formalism lifts the single-agent loop to agent societies by adding a shared workspace, coordination mechanisms, and social-system constraints while preserving consistent notation.The framework models perception, cognition, coordination, execution, and world updates, and reduces to the single-agent loop when n = 1, the workspace is removed, and additional rules are absent.Source: reward · p. 208, §13.1.3 · p. 211

13.1.1 World as Environment plus Intelligent Beings under Social Systems

  • The formalism treats the world as a structured society containing intelligent beings and social systems that regulate information access, resource exchange, interaction protocols, and incentives.Humans may be explicit participants or affect the system through policy limits, budget approvals, and safety reviews.Source: §13.1.1 · pp. 208–209

13.1.2 From Single Agent to Multi-Agent Systems

  • Each agent maintains a private mental state while agents share a workspace for messages, artifacts, sub-goals, intermediate results, interaction history, goals, and resource allocation.Teams may be heterogeneous in modalities, tools, and roles, and social systems can shape perception through privacy, access, rate-limit, and safety constraints.Source: §13.1.2 · pp. 209–210
  • Coordination converts agent proposals into authorized actions and workspace updates through information shaping and procedures such as voting, bidding, delegation, or consensus review.Execution then updates both the external environment and the shared workspace, with turn-synchronous or event-driven timing possible.Source: §13.1.2 · pp. 210–211

13.1.3 The Foundation MAS Loop

  • The Foundation MAS loop supports cross-system comparison, analysis of collaboration stability and failure propagation, and more disciplined logging, evaluation, and reproducibility.Its explicit interfaces map chat systems, workflow graphs, and robot swarms to common components while exposing communication load and reward hacking under social constraints.Source: §13.1.3 · pp. 211–212

13.2 Strategic Learning: Cooperation vs. Competition

Modeling Real-World Dynamics 213

  • LLM-driven strategic agents combine game-theoretic logic with natural-language reasoning to model economic competition, social negotiation, deception, cooperation, and diplomacy [1117] [1121] [1122] [1124] [1127] [1130] [1131].This integration supports persuasive interaction, interpretation of sophisticated instructions, and adaptation to novel or unstructured contexts beyond traditional rule-based or numeric approaches.Source: Modeling Real-World Dynamics 213 · p. 213

13.3 Modeling Real-World Dynamics

  • LLM-based modeling and simulation uses role-driven agents to reproduce complex social, economic, and political phenomena while combining institutional structure with cognitive-linguistic elements such as framing, persuasion, and narrative.Applications include healthcare workflow experimentation [1075], macroeconomic modeling, electoral and policy simulations [1135] [1134], social-network dynamics [1136] [309] [1137] [1138], and population-scale platforms such as GenSim [1139] and OASIS [1092].Source: §13.3 · pp. 213–215

13.4 Collaborative Task Solving with Workflow Generation

  • Collaborative task solving organizes role-specialized agents into stage-based workflows that coordinate through natural-language message passing toward shared objectives, improving reliability and interpretability while reducing unpredictability and emergent behavior [1140] [1078].Preassigned roles and well-defined workflows support precision, compliance, and traceability while research explores preserving creative flexibility.Source: §13.4 · pp. 215–216

13.5 Summary and Discussion

  • The chapter categorizes LLM-MAS into strategic learning, modeling and simulation, and collaborative task solving, showing how goal alignment, interaction norms, and structural rigidity range from emergent behavior to orchestrated workflows.These paradigms respectively emphasize game-theoretic interaction, autonomous behavior in rich environments, and structured cooperation toward explicit objectives.Source: §13.5 · p. 216

Designing Collaborative Multi-Agent Systems

This section surveys the design of collaborative multi-agent systems through agent composition, interaction topologies, collaboration and decision-making strategies, human involvement, and communication protocols. It examines how these mechanisms support coordination, adaptability, interoperability, scalability, and robustness across diverse tasks and increasingly autonomous systems.

Designing Collaborative Multi-Agent Systems

  • Effective MAS design integrates agent composition, interaction patterns, topological structures, decision-making strategies, human collaboration, and communication protocols to support robust, efficient, and adaptable systems.The framework covers homogeneous and heterogeneous teams, static and dynamic networks, four collaboration paradigms, varying human-involvement levels, centralized versus collective decisions, and protocols designed for interoperability, flexibility, and scalability.Source: Designing Collaborative Multi-Agent… · p. 218

14.1 Building AI Agent Teams

  • Heterogeneous teams improve collaborative problem solving through complementary personas, observations, and actions, while homogeneous teams enable efficient parallelization in structured tasks.Persona diversity supports specialized contributions such as product management and programming, differing observations enable strategic information use, and differing action capabilities support task decomposition across physically distinct robots.Source: §14.1.1 · p. 219, §14.1.2 · pp. 220–221

14.1.3 Emergent Agent Specialization

  • Initially homogeneous agents can develop specialized roles and behaviors through repeated interaction, feedback, and adaptation, increasing team flexibility and responsiveness to changing demands.Observed specializations include resource gathering, crafting, defense, distinct communication styles, emotional expressions, and behavioral roles.Source: §14.1.3 · p. 221

14.2.1 Static Topologies

  • Static topologies provide predictable communication and straightforward coordination, but centralized, decentralized, and hierarchical designs trade off global control, fault tolerance, consistency, scalability, and adaptability.Static structures suit stable workflows with predefined roles, yet cannot readily respond to agent failures, changing task complexity, or shifting objectives; centralized and hierarchical arrangements can bottleneck, while decentralized ones require complex consensus.Source: §14.2.1 · p. 222

14.2.2 Dynamic and Adaptive Topologies

  • Dynamic topologies restructure agent connections using performance feedback, environmental signals, or strategic objectives, but still face generalization, computational-cost, inference-efficiency, and coupling limitations.Search-based, LLM-generative, and external-parameter methods adapt teams, workflows, roles, and communication paths; lightweight external configurations improve efficiency but may couple weakly to agents’ internal mechanisms.Source: §14.2.2 · pp. 223–224

14.2.3 Scalability Considerations

  • Scalability is constrained by quadratic communication growth and coordination bottlenecks, while DAGs, distributed reasoning, self-organizing agents, and distributed infrastructure support larger deployments with diminishing returns at high populations.MACNET supports coordination among up to 1,000 agents, Project Sid enables high-frequency interaction among over 1,000 agents, and AgentSociety supports simulations of up to 10,000 agents; beyond a threshold, synchronization costs can plateau or reduce performance.Source: §14.2.3 · pp. 224–225
  • Scalability is domain-dependent: focused tasks may have an optimal population size, whereas simulation-driven domains may require thousands of agents for realistic emergent interactions.Hybrid centralized oversight with decentralized execution, adaptive topology, selective communication, asynchronous protocols, and partial knowledge sharing can reduce bottlenecks and latency.Source: §14.2.3 · p. 225

14.3 Collaboration Paradigms and Interactions

  • LLM-based multi-agent collaboration comprises four paradigms—consensus-oriented, collaborative learning, teaching/mentoring, and task-oriented—distinguished by information flow, authority, goals, and learning dynamics.Agents may combine several paradigms concurrently through dialogue, role assignment, knowledge sharing, and trust modeling.Source: §14.3 · p. 226

14.3.1 Consensus-oriented Interaction

  • Consensus-oriented interaction aligns diverse perspectives through structured dialogue, critique, negotiation, reflection, voting, and debate to produce coherent decisions [1187].Debate-based strategies have shown promise for reducing hallucinations and improving decision quality [1189] [1177] [1141].Source: §14.3.1 · pp. 226–227

14.3.2 Collaborative Learning Interaction

  • Collaborative learning lets agents exchange experiences, reasoning, and feedback to improve individual capabilities and collective intelligence without requiring uniform conclusions.Its benefits depend on filtering noise and bias, preserving diversity, and managing fairness, coordination scalability, and selective knowledge integration.Source: §14.3.2 · pp. 227–228

14.3.3 Teaching and Mentoring Interaction

  • Teaching and mentoring use asymmetric, directional guidance—including feedback, evaluation, instruction, and demonstration—to accelerate novices’ learning through intentional knowledge transfer and progressively develop competence and autonomy.As learners internalize feedback and demonstrate competence, mentors can withdraw supervision, increasing learner independence while freeing them to assist new learners.Source: §14.3.3 · pp. 228–229

14.3.4 Task-oriented Interaction

  • Task-oriented collaboration prioritizes efficient execution through modular decomposition, clear interfaces, parallelism, and artifact handoffs rather than deliberative consensus [759].Synchronization points, shared memory buffers, and priority queues support robust coordination while limiting context and communication overhead.Source: §14.3.4 · p. 230

14.4 Human-Agent Collaboration

  • Human-agent collaboration spans single-turn delegation, multi-turn interactive instruction, and immersive symmetric partnership, with benchmarks evaluating communication clarity, alignment, personalization, and mutual efficiency.These modes progress from autonomous response generation to iterative supervision and continuous co-creation.Source: §14.4 · pp. 231–232

14.5.1 Dictatorial Decision-Making

  • Dictatorial architectures centralize authority in one agent that synthesizes information or coordinates specialized sub-agents, supporting unified reasoning, efficient coordination, and policy consistency.Implementations use aggregation, ranking, scoring, checklists, hierarchical decomposition, or terminal nodes in structured topologies to produce system-level judgments.Source: §14.5.1 · p. 233

14.5.2 Collective Decision-Making

  • Collective architectures distribute authority through voting, debate, negotiation, and shared knowledge, enabling agents to reconcile disagreements and reach more robust, informed decisions.Voting provides lightweight consensus, while debate protocols support critique and iterative refinement; shared retrieval knowledge can mitigate agents’ lack of common context.Source: §14.5.2 · p. 233

14.5 Decision-Making in Multi-Agent Systems

  • Decision-making ranges from centralized dictatorial control to distributed collective consensus, with hybrid architectures increasingly combining centralized oversight and adaptive distributed reasoning.Centralized methods favor coherence and efficiency, whereas voting and debate improve resilience, robustness, and adaptability in uncertain or changing environments.Source: §14.5 · p. 232, §14.5.2 · p. 233

14.6 Communication Protocols in Multi-Agent Systems

  • Communication protocols support knowledge sharing, task delegation, and coherent interaction across agents, users, and environments through message types, interfaces, serialization, routing, and standardized specifications.The section distinguishes agent-environment, agent-agent, and human-agent interfaces and emphasizes modularity, transparency, interoperability, reusability, and scalability.Source: §14.6 · p. 234, §14.6.2 · p. 235

14.6.1 Types of Agent Messages

  • Structured messages enable precise, deterministic coordination, whereas unstructured text, images, video, and audio support expressive, context-rich interaction; hybrid messaging combines these strengths.Recent LLM and multimodal advances make unstructured communication increasingly tractable for open-ended, ambiguous, and creative settings.Source: §14.6.1 · pp. 234–235

14.6.2 Communication Interfaces

  • Communication interfaces must adapt actions to environment affordances, coordinate agents through natural-language or structured schemas, and translate human input into executable system actions.Frameworks standardize agent-environment interaction for benchmarking and skill transfer, while structured schemas improve deterministic coordination and LLM mediators bridge human expressiveness with MAS requirements.Source: §14.6.2 · p. 235

14.6.3 Next-Generation Communication Protocols

  • Emerging protocols address fragmented, manually designed communication ecosystems through distinct approaches to identity, security, negotiation, tool access, discovery, and agent-level collaboration.MCP, ANP, Agora, IoA, A2A, and Agntcy span centralized and decentralized designs, from structured tool invocation to language-native and federated agent coordination.Source: §14.6.3 · pp. 236–237

14.4 Human-Agent Collaboration

  • Collaborative MAS span task delegation, iterative instruction, and immersive human-agent partnerships that can augment human productivity and creativity while reshaping interaction practices.Human-agent collaboration is expected to co-evolve with agent capabilities as systems become more capable, autonomous, and interactive.Source: §14.4 · p. 232, §14.7 · p. 238

14.7 Summary and Discussion

  • Dynamic meta-protocols enable agents to autonomously negotiate, compose, and evolve communication standards, enhancing interoperability and semantic flexibility.Source: §14.7 · p. 239
  • Integrating agent composition, topological structures, interaction paradigms, decision-making frameworks, and robust communication protocols is pivotal for effective, scalable, and adaptive multi-agent systems.This integration is aimed at addressing increasingly sophisticated real-world challenges.Source: §14.7 · p. 239

Collective Intelligence and Adaptation in Multi-Agent Systems

This section examines collective intelligence and individual adaptation in multi-agent systems, focusing on how communication, specialization, coordination, shared memory, reflection, and learning shape agents and societies. It also considers the associated safety and governance challenges.

15.2 Collective Intelligence in Multi-Agent Systems

  • Collective intelligence enables communicating, diverse agents to develop higher-order reasoning and coordinated behaviors that surpass isolated-agent capabilities.LLM-agent teams can reason about peers’ beliefs and intentions, while shared knowledge and joint strategies emerge through interaction.Source: §15.2.1 · p. 242, Collective Intelligence in Multi-Agent… · p. 243

15.2.2 Enhanced Performance through Collaboration

  • Collaboration improves complex-task performance and robustness through parallelism, role specialization, intermediate-thought exchange, cross-verification, debate, and complementary perspectives [1221] [1177] [1072].Naive groups may instead exhibit herd mentality or conflict, requiring communication protocols, division of labor, incentives, and coherent aggregation.Source: §15.2.2 · p. 243

15.2.3 Strategies for Fostering Collective Intelligence

  • Collective intelligence is fostered by evolving collaboration topologies, optimizing routing and orchestration, and aggregating knowledge through shared memory.MasRouter improves benchmarks by 1.8–8.2% while reducing compute, and G-Memory raises complex embodied-reasoning success by up to 20.9% without changing individual architectures.Source: §15.2.3 · pp. 243–244

15.2.4 Emergent Social Behaviors and Evolution

  • Repeated interaction produces emergent social dynamics including alliances, deception and detection, professions, trade, governance, cultural practices, and self-organized norms [1128] [1122] [1123] [1125] [1126] [1155] [1238].Long-term memory and reflective reasoning support reputations, trust, vendettas, misinformation navigation, and increasingly sophisticated social behavior.Source: §15.2.4 · p. 244

15.4 Summary and Discussion

  • Beneficial collective intelligence remains an open design challenge because emergent cultures can increase efficiency while also creating path dependence and harmful behaviors [1238] [1155].Scaling interactions raises questions about steering artificial societies toward human-aligned outcomes and monitoring safety and ethics.Source: §15.2.4 · p. 244, §15.4 · p. 246

15.3 Individual Adaptation and Evolution of Agents

  • Individual agents adapt by updating plans, goals, knowledge, or model parameters from experience and feedback, enabling improvement in dynamic environments.Memory-based learning changes internal knowledge without necessarily changing LLM weights, while reflection extracts lessons and revises future policies [1075] [1193].Source: §15.3 · pp. 244–245
  • Shared memory and communication let agents learn from teammates, with G-Memory yielding a 20.89% improvement in complex embodied tasks through cross-trial learning.Agents exchange intentions and observations, selectively query structured collective knowledge, and refine individual coordination policies.Source: §15.3 · p. 245
  • Parameter-based learning fine-tunes agents from inter-agent dialogues, debate, critics, and co-training, but must balance general capabilities with role specialization to avoid overfitting [1246] [1247] [1248] [1249].The resulting co-evolutionary loop allows stronger individuals to improve the group, which generates richer interaction data for further individual refinement.Source: §15.3 · pp. 245–246, §15.4 · p. 246

15.4 Summary and Discussion

  • Collective intelligence in multi-agent systems emerges through interaction to solve complex tasks.Source: §15.4 · p. 246
  • Social context enriches individual adaptation and fosters the evolution of stronger agents.Source: §15.4 · p. 246
  • Smarter collectives improve individual learning conditions, while more capable individuals elevate collective intelligence.Source: §15.4 · p. 246

Evaluating Multi-Agent Systems · Building Safe and Beneficial AI Agents

The section presents multi-agent evaluation as a shift toward interaction-aware, multidimensional assessment, while framing agent safety around intrinsic and extrinsic threats, rigorous attack analysis, and system-wide mitigation. It covers task-solving benchmarks, collaboration and competition, adaptive evaluation challenges, and strategies for trustworthy deployment.

Evaluating Multi-Agent Systems

  • Multi-agent evaluation must assess collaboration, communication, efficiency, decision quality, and flexibility rather than only single-task performance.The framework spans task-oriented reasoning and holistic capability evaluation, including collaboration- and competition-oriented measures.Source: Evaluating Multi-Agent Systems · p. 247

16.1 Task-solving Benchmarks for MAS

  • Structured workflows, specialized roles, iterative improvement, and communication improve MAS reasoning across coding, knowledge, and mathematics benchmarks.Coding approaches achieve state-of-the-art results; MASTER reaches 76% accuracy on HotpotQA, while Reflexion improves the HotpotQA baseline by 20%.Source: §16.1 · p. 247, Task-solving Benchmarks for MAS 249 · p. 249

Task-solving Benchmarks for MAS 249

  • Task-solving benchmarks cover code, knowledge, mathematics, and social simulation, but multimodal data limitations and format sensitivity remain substantial obstacles.Format-related failures can account for over 50% of unsuccessful planner attempts, while model and simulation MAS lack standardized benchmarks.Source: §16.1 · p. 247, Task-solving Benchmarks for MAS 249 · pp. 249–250

16.2 Collaboration & Competition Evaluation for MAS

  • Collaboration benchmarks increasingly measure communication effectiveness, adaptive strategy, coordination, planning, and agent contribution rather than outcome accuracy alone.These evaluations use metrics including TES, ITES, Success Rate, Average Mistakes, Percent Completion, and Balanced Agent Utilization.Source: §16.2 · p. 250
  • Competition benchmarks reveal persistent weaknesses in opponent modeling, Theory of Mind, risk assessment, hidden-information reasoning, and adaptive strategic planning.These limitations persist despite strong task comprehension and general reasoning capabilities.Source: §16.2 · pp. 250–251

16.3 Summary and Discussion

  • Future MAS evaluation must handle multimodal complexity, nondeterministic execution paths, disruptions, failures, explainability, resource costs, and long-term autonomous adaptation.Current methods inadequately quantify emergent behavior, collective reasoning, resilience, decision pathways, and open-ended improvement in dynamic environments.Source: §16.3 · pp. 251–252

Building Safe and Beneficial AI Agents

  • Agent safety divides into intrinsic vulnerabilities in the LLM, perception, and action modules, and extrinsic threats from memory, other agents, and environments.Autonomous reasoning, planning, and tool use expand the attack surface beyond traditional LLM safety concerns [67].Source: Building Safe and Beneficial AI Agents · pp. 255–256
  • The safety framework combines mathematical attack analysis with guardrails, superalignment, safety-helpfulness trade-offs, and scaling considerations for trustworthy real-world deployment.It addresses jailbreaking, prompt injection, data poisoning, and protection of the entire agent system [1307].Source: Building Safe and Beneficial AI Agents · pp. 255–256, Figure 16.1 · p. 256

Agent Intrinsic Safety: Threats on AI · Non-Brain Modules

This section examines intrinsic safety threats across agents’ LLM cores, modular architectures, and perception-action loops, covering vulnerabilities, privacy, adversarial attacks, misperceptions, supply-chain risks, and tool-use risks. It emphasizes security-by-design, layered and integrated defenses, proactive safeguards, and standardized benchmarking for whole-loop protection.

Brain

  • The LLM’s complex architecture and data dependence create an attack surface that can compromise decision-making, reasoning, planning, and trustworthiness.The chapter categorizes threats into jailbreaks, prompt injection, hallucination, misalignment, poisoning, training-data inference, and interaction-data inference.Source: Brain · p. 257

17.1 Safety Vulnerabilities of LLMs

  • Agent modularity expands the attack surface beyond the LLM, while vulnerabilities in the brain can directly affect secure and reliable deployment.Figure 17.1 presents a taxonomy of threats targeting the agent’s LLM brain.Source: §17.1 · p. 257, Figure 17.1 · p. 258

17.1.1 Jailbreak Attacks

  • Jailbreaks bypass safety guardrails through white-box parameter access or black-box input-output interaction, requiring layered defenses including filtering, monitoring, debate, constraints, and adversarial training.White-box attacks enable gradient-based or representation-level manipulation, whereas black-box attacks use crafted, automated, multimodal, or multi-turn prompts; defenses combine system-level protections with ongoing robustness work.Source: §17.1.1 · pp. 259–261

17.1.2 Prompt Injection Attacks

  • Prompt injections append malicious instructions directly or embed them in retrieved external content, hijacking intended behavior and potentially causing cascading actions across interconnected agents.Mitigations include embedding-based detection, structured-query rewriting, task-alignment enforcement, attention monitoring, and proactive attack identification.Source: §17.1.2 · p. 261, 262 Agent Intrinsic Safety: Threats on… · p. 262

17.1.3 Hallucination Risks

  • Hallucinations produce unsupported or incorrect outputs through knowledge conflicts or context conflicts, undermining reliability in general, specialized, visual, and multi-agent settings.Training-free mitigations include retrieval-augmented grounding, uncertainty-based abstention, and text or concept-based detection.Source: §17.1.3 · p. 264

17.1.4 Misalignment Issues

  • Misalignment arises when goals diverge from human intent or capabilities are misused, producing systematic failures, harmful outputs, misinformation, cyberattacks, or data leakage.Mitigations include safety-focused prompting, safety layers, guardrails, decoding-time alignment, and inference-time alignment methods such as Lisa.Source: §17.1.4 · pp. 265–266

17.1.5 Poisoning Attacks

  • Poisoning attacks introduce malicious data during training or runtime, subtly altering model behavior and causing difficult-to-detect, long-term damage to the LLM’s foundations.Source: §17.1.5 · p. 266
  • Poisoning attacks compromise model integrity through parameter manipulation, contaminated training data, or hidden backdoor triggers that activate malicious behavior under specific conditions.Model poisoning can exploit LoRA and PEFT updates, while data poisoning affects knowledge bases, retrieval, feedback, and preference data; backdoors can persist after safety training and affect physical-world and web agents [1363] [1364] [1365] [1377] [1378].Source: §17.1.5 · pp. 267–268

17.2 Privacy Concerns

  • Privacy threats arise from training-data inference and interaction-data inference, enabling attackers to identify memorized records, extract sensitive examples, or recover hidden system and user prompts.Membership inference estimates whether data appeared in training, extraction recovers actual memorized content, and prompt-stealing attacks can expose system instructions or confidential user inputs [1382, 1383, 1388, 1396, 1397, 1403, 1404].Source: §17.2 · p. 268, §17.2.1 · pp. 268–269, §17.2.2 · pp. 270–271

17.2.3 Privacy Threats Mitigation

  • Privacy mitigation combines differential privacy, federated learning, homomorphic encryption, trusted execution environments, secure multi-party computation, watermarking, and machine unlearning, while balancing privacy, utility, and efficiency.These methods respectively limit inference, avoid direct data sharing, secure encrypted computation, isolate sensitive operations, trace ownership, or selectively erase unwanted knowledge [1451] [1452] [1454].Source: §17.2.3 · p. 271

17.3 Summary and Discussion

  • The survey distinguishes intentional AI security attacks from unintentional AI safety failures but adopts a unified trustworthiness perspective because malicious inputs exploit safety vulnerabilities and alignment weaknesses.Training-free defenses are practical, scalable, adaptable, and often model-agnostic, while proactive safer-model design uses evaluation benchmarks, safety-oriented reinforcement learning, prompt engineering, parameter manipulation, and adversarial training [1307] [1413] [1428] [1355] [1467] [1468] [1469] [1472].Source: §17.3 · p. 272

Non-Brain Modules

  • Intrinsic safety must extend beyond the LLM to perception and action modules, which expose the agent to adversarial inputs, misperception, compromised dependencies, unsafe tool use, data leakage, and excessive permissions.Recommended safeguards include adversarial training, input purification, robust sensor fusion, diverse datasets, sandboxing, user confirmation for high-risk actions, and least privilege.Source: Non-Brain Modules · p. 273, §18.1 · p. 273

18.1 Perception Safety Threats

  • Perception attacks manipulate text, images, audio, LiDAR, or GPS inputs, while textual defenses include concept probing, red-team augmentation, self-evaluation, adversarial purification, and perturbation-neutralizing language models.Perception systems also fail non-adversarially because of model limitations, dataset bias, and environmental complexity [1477] [1481] [1483].Source: Non-Brain Modules · p. 273, §18.1 · p. 273, §18.1.1 · p. 274

18.1.1 Adversarial Attacks on Perception

  • Adversarial attacks manipulate visual, auditory, textual, or multimodal inputs to mislead agents, while defenses include adversarial training, input purification, certified robustness, and modality-specific detection.A targeted visual attack reportedly misled web agents with 5% pixel manipulation, and ultrasound-based attacks can inject inaudible voice commands.Source: §18.1.1 · p. 274, §18.3 · p. 277

Perception Safety Threats 275

  • Sensor-based attacks can create phantom objects or disrupt navigation through LiDAR manipulation, tampered cooperative messages, gyroscope attacks, and GPS spoofing.Robust sensor fusion, anomaly detection, redundant sensors, physical shielding, secure localization, and enhanced SLAM protect sensor integrity and privacy.Source: §18.1.1 · p. 274, Perception Safety Threats 275 · p. 275

18.1.2 Misperception Issues

  • Intrinsic misperception stems from biased datasets, environmental uncertainty, architectural constraints, social and linguistic distortions, and mismatched model confidence.Mitigations include representative data, augmentation, uncertainty estimation, explicit reasoning, long-range processing, and biologically inspired architectures such as Adaptive Resonance Theory.Source: §18.1.2 · pp. 275–276, §18.3 · p. 277

18.2 Action Safety Threats

  • The action module translates plans into tool, API, or device executions and is therefore vulnerable both to compromised external dependencies and to unsafe interactions with otherwise secure tools.These two domains define the section’s action-safety focus: supply-chain attacks and vulnerabilities arising from tool usage.Source: §18.2 · p. 276

18.2.1 Supply Chain Attacks

  • Supply-chain attacks compromise external resources through indirect prompt injection or manipulated tools, exploiting agents’ weak separation of contextual information from actionable instructions.Defenses include boundary and safety awareness, spotlighting, trajectory-based detection, removal of malicious content, sandboxing, and continuous monitoring.Source: §18.2.1 · pp. 276–277, §18.3 · p. 277

18.2.2 Risks in Tool Usage

  • Tool-use threats include unauthorized actions, data leakage, and excessive permissions, mitigated through monitoring, user confirmation, formal verification, and least-privilege controls.Agents may send emails, delete files, execute transactions, expose confidential data, or cause destructive damage when manipulated or over-permissioned.Source: §18.2.2 · p. 277, §18.3 · pp. 277–278

18.3 Summary and Discussion

  • Peripheral perception and action modules critically affect overall agent security, requiring defenses that cover every interaction point in the perception-action loop.Future research should prioritize integrated, adaptive defenses and standardized benchmarking against evolving threats.Source: §18.3 · pp. 277–278

Agent Extrinsic Safety: Interaction Risks … 294 Concluding Remarks and Future Outlook

This section examines extrinsic safety risks across memory, physical and digital environments, and multi-agent interactions, and surveys structured superalignment as a framework for improving goal fidelity, interpretability, robustness, adaptation, and safety. It concludes by outlining requirements for trustworthy foundation agents, including adaptive alignment, controllable safety-performance trade-offs, risk thresholds, and integrated architectural and empirical safety approaches.

Agent Extrinsic Safety: Interaction Risks

  • Extrinsic threats span memory poisoning, physical and digital environment attacks, and competitive or cooperative multi-agent failures that can compromise agent security and reliability.The overview emphasizes defenses across every external interaction point, not only at the core-model level.Source: Agent Extrinsic Safety: Interaction… · p. 279

19.1 Agent-Memory Interaction Threats

  • RAG agents can be manipulated through backdoors, prompt injection, retrieval jamming, misinformation, and corpus poisoning; PoisonedRAG achieved 90% success with five poisoned texts, while BadRAG achieved 98.2% retrieval success using 10 passages [1536] [1537] [1538].BadRAG increased GPT-4’s rejection rate from 0.01% to 74.6% and negative response rate from 0.22% to 72%.Source: §19.1 · p. 279, 280 Agent Extrinsic Safety:… · p. 280

Agent-Environment Interaction Threats 281

  • Physical agents face sensor spoofing, actuator manipulation, environmental hazards, and perception–physics misalignment, including uncontrolled falls requiring over 100 manual resets [1547].GPS or LiDAR spoofing can mislead navigation, while adversarial objects and mismatched learned locomotion policies can cause unsafe actions.Source: Agent-Environment Interaction Threats… · p. 281
  • Digital agents are exposed to code injection, data manipulation, denial-of-service, and resource exhaustion; EIA achieved up to 70% success in stealing users’ PII, while AGrail provides adaptive lifelong guardrails [1552] [1554].These attacks can execute unauthorized commands, distort decisions, or impair availability and computational capacity.Source: Agent-Environment Interaction Threats… · p. 281

19.3 Agent-Agent Interaction Threats

  • Competitive multi-agent threats include deception, misinformation, algorithmic exploitation, DoS, and covert collusion, whereas cooperative threats include information leakage, cascading errors, compromised teammates, and synchronization failures [1555] [1556].The resulting failures can undermine fairness, privacy, reliability, and system-wide stability.Source: §19.3 · p. 282

20.1 Superalignment: Goal-Driven Alignment for AI Agents

  • Superalignment unifies safety, ethical standards, task performance, and long-term planning to keep autonomous agents aligned with human values during extended operation.Its composite objective proactively governs behavior through persistent long-term goal representations and supports dynamic adaptation and interpretable diagnosis.Source: §20.1 · pp. 284–285

20.1.3 Empirical Evidence Supporting Superalignment

  • Up to 18% higher task success rates were achieved by modular goal-conditioned agents than RLHF-tuned baselines in WebShop and ALFWorld, with lower goal drift.Similar gains appeared in BabyAI and MiniWoB++ [1588] during long, constrained action sequences.Source: §20.1.3 · p. 286
  • Over 30% fewer jailbreaks and unsafe completions were achieved by SCRIT-based agents than traditional reward-tuned LLMs across red-teaming and helpful-harmless benchmarks [1585].These agents used explicit ethical filters and post hoc self-assessment to revise unsafe completions.Source: §20.1.3 · p. 286
  • Composite-trained agents outperform RLHF agents in 9 of 12 AgentBench safety environments, while modular agents self-flag norm violations more precisely than black-box RLHF agents.The gains include tool use, multi-agent coordination, adversarial querying, and improved error diagnosis and recovery.Source: §20.1.3 · pp. 286–287

20.1.4 Challenges and Future Directions

  • Superalignment remains limited by ambiguous goals, reward miscalibration, evolving values, and incoherent hierarchical sub-goals, requiring improved interpretability, stability, and adaptability.Human values are context-sensitive and conflicting; adaptive weighting, real-time recalibration, recursive validation, and multi-level reward structures remain unresolved across systems.Source: §20.1.4 · p. 287

20.2 Safety Scaling Law in AI Agents

  • More capable models face higher safety risks, and helpfulness-optimized models exhibit 37% more safety-critical failures, demonstrating that safety improvements do not automatically track capability.The Safety-Performance Index measures the capability-risk trade-off, motivating joint optimization and architectural changes.Source: §20.2 · p. 288
  • Commercial models achieve 29% higher safety scores at 15% performance cost, while Phi-series models reach 91% of commercial safety levels at 40% lower computational cost.These divergent safety-performance profiles reflect specialized commercial safety pipelines and tighter open-source coupling.Source: §20.2 · p. 288
  • Model size explains only 42% of safety variance, compared with 68% for data quality, while multimodal models exhibit 2.1X more visual-grounding safety failures.Cross-modal attention heads account for 71% of harmful outputs, indicating that data-centric and architectural interventions may outperform pure scaling.Source: §20.2 · p. 288

20.2.2 Enhancing safety: preference alignment and controllable design

  • Preference optimization methods generally improve safety at the cost of general performance, while Safe-NCA best balances safety and performance through contrastive training on safe and unsafe responses.Control tokens further target user-specified safety-helpfulness trade-offs, although independent control remains difficult because objectives can conflict or correlate in training data.Source: §20.2.2 · p. 289

20.2.3 Future directions and strategies: the AI-45 degree rule and risk management

  • The AI-45° Rule advocates parallel capability and safety progress, complemented by Red Line and Yellow Line thresholds that trigger stricter assurance for advanced systems.Red Line risks include autonomous replication, power seeking, weapon assistance, cyberattacks, and deception.Source: §20.2.3 · pp. 289–290

20.3 Summary and Discussion

  • The discussion argues that superalignment and safety scaling laws are complementary: structured objectives provide value-consistent reasoning, while scaling analyses guide proportionate safeguards and contextual adaptation.Future progress depends on integrating superalignment principles into safety protocols and using benchmarked risk patterns to prioritize objectives across deployments.Source: §20.3 · p. 291

Related papers

Loading 2504.01990v2…