Bringing AI to Autonomous Systems -- From Cognition to Collective Intelligence

1. Introduction

• AI provides building blocks for intelligent systems, but not the engineering principles needed to design them reliably or realize autonomous systems with little human intervention.Autonomous systems coordinate agents pursuing individual goals while contributing to overall system goals.Source: §1 · p. 1, Autonomous Systems: The Ultimate Stage · p. 2

The Gap Between Vision and Reality

• Current AI agents remain low-reliability and primarily digital, while safety-critical tasks require failure rates around 10^-8 per operating hour, beyond current AI components.The paper contrasts this gap with predictions of fully autonomous vehicles by 2020 that proved wrong.Source: The Gap Between Vision and Reality · p. 2

The Historical Context

• Multi-agent systems have roots in 1980s symbolic AI, but current protocols provide communication channels without resolving what agents should communicate, why, or how to make interactions reliable.Source: The Historical Context · p. 2

The Trustworthiness Challenge

• AI’s opaque, data-learned behavior changes what reliability means and what evidence can demonstrate it; current studies assess cognitive properties using behavioral criteria.The paper argues that cognitive properties also involve intention, accountability, and understanding and respecting behavioral norms, conditions it says current AI does not meet.Source: The Trustworthiness Challenge · pp. 2–3

Contributions of This Article

• The article combines connectionist AI, symbolic AI, and systems engineering in a framework organizing cognitive functions around long-term memory and addressing perception-to-memory links, goal planning, and agent coordination.Source: Contributions of This Article · p. 3
• Agent trustworthiness includes a cognitive dimension whose validity depends on how knowledge informs decisions, and the article outlines avenues to evaluate it while assessing the gap between vision and current capabilities.Source: Contributions of This Article · p. 3

2.1 The Trend Toward Architectural Solutions

• Machine-learning breakthroughs enabled machines to analyze complex unstructured data, but agents still need an architecture connecting perception with understanding, reasoning, decision-making, planning, and action.Source: §2.1 · pp. 3–4

The Emerging Trend Toward Modular Architectures

• The proposed architecture follows a shift from monolithic systems toward modular functions, with agents needing knowledge of a changing world and the ability to anticipate their environment and plan toward goals.It identifies agent components and their relationships through a knowledge base.Source: §2.1 · p. 4, The Emerging Trend Toward Modular… · p. 5

Three Approaches to Endowing Agents with Reasoning

• Reasoning approaches include scaling models, combining neural networks with symbolic engines, and learning world models; prediction alone cannot choose actions, resolve value conflicts, or provide normative reasoning.The symbolic approach faces the challenge of reliably linking knowledge derived from data with symbolic knowledge.Source: Three Approaches to Endowing Agents… · p. 4

2.2.1 The Agent's Mission: What is the agent supposed to achieve?

• An agent pursues transient goals while continuously maintaining invariant goals, planning around environmental responses; the breadth of its goals shapes its degree of autonomy.Invariant goals prevent undesirable outcomes, while transient goals direct the agent toward targets; both are needed for progress without sacrificing constraints such as safety.Source: §2.2.1 · pp. 6–7

2.2.2 The Agent's Faculties: The Tools of Agency

• Agent faculties comprise capabilities, competencies, aptitudes, and authority, respectively defining feasibility, reasoning, social interaction, and institutional permissions.Capabilities may include physical or certified abilities; authority specifies an agent’s roles, privileges, and power relationships within the system.Source: §2.2.2 · pp. 7–8

2.2.3 Fitting In: Integration Conditions

• Integration conditions combine inviolable physical constraints, enforceable norms, and value systems that weigh utility, preferences, costs, benefits, and risks.Norms distinguish permitted, required, or prohibited actions; value systems distinguish what is preferred, helping resolve conflicts between goals and actions.Source: §2.2.3 · pp. 8–9

2.2.4 Staying Alive: The Agent's Operating Conditions

• Operating conditions are internal prerequisites for mission achievement, spanning survival, safety and security, self-fulfilment, and social recognition; violations call for proactive remedial goals.These conditions include resources such as energy and computation, integrity against failures or attacks, adapting capabilities through learning, and acceptance by other agents.Source: §2.2.4 · pp. 9–10

The Autonomous Driving Agent

• The autonomous driving agent’s mission is safe, efficient passenger transport, supported by relevant knowledge, physical capabilities, environmental constraints, and internal operating conditions.Its capabilities and certified scope bound possible actions; it must account for physics, traffic laws, passenger expectations, and conditions such as battery level and sensor health.Source: The Autonomous Driving Agent · p. 11

The Communicator Agent

• The Communicator agent exchanges information reliably and accurately, using linguistic faculties and trust-related knowledge while observing communication norms and maintaining operational resources.Its competencies include detecting deception, negotiating, and building consensus; its internal conditions include buffer space, connectivity, and processing capacity.Source: The Communicator Agent · pp. 11–12

Summary

• Driving capabilities are physical and governed by physics and traffic laws, while communicator capabilities are linguistic and governed by protocols and standards.Despite these differences in attribute values and integration conditions, both agents can share the same generic architecture.Source: Summary · p. 12

2.3 Agent Reference Architecture: Specification and Behavior

• The architecture links three coordinated modules through a knowledge base that guides their decisions and is updated to reflect them.It is a theoretical reference model, not a prescribed implementation.Source: §2.3 · pp. 12–13

The Perception Module

• Perception turns raw data into a coherent World-state representation, using the knowledge base to focus on relevant information and check consistency.Inconsistencies are corrected or trigger an alert before the World state is updated.Source: The Perception Module · p. 13

The Reactive Module

• Reactive behavior models possible futures from the perceived World state, then selects and executes a goal-directed plan constrained by capabilities, norms, and values.The plan alternates controllable and uncontrollable actions; the first controllable action is executed.Source: The Reactive Module · p. 14

The Proactive Module

• The proactive module monitors internal operating conditions, generates meta-goals when a condition fails, and selects a feasible, permitted, high-utility goal for reactive execution.Meta-goals may be pre-calculated, rule-defined, or determined through machine learning.Source: The Proactive Module · p. 14

Reactive Behavior vs. Proactive Behavior

• Reactive behavior responds to the World, while proactive behavior maintains the Self; both are essential for long-term autonomy.Without proactive behavior an agent risks resource exhaustion, poor adaptation, or loss of other agents’ trust; without reactive behavior it cannot respond to environmental opportunities and threats.Source: Reactive Behavior vs. Proactive Behavior · p. 15

2.4.1 How the Self-Driving Agent Works

• The self-driving agent follows the reference architecture’s three-process pattern through concrete physical functions.Its perception processes camera, radar, and lidar data against knowledge-base constraints before updating the World state.Source: §2.4.1 · p. 15, Perception: Interpreting the World · p. 15

Perception: Interpreting the World

• The car’s perception checks sensory data for consistency with the knowledge base, then updates the World state with validated information.For a red light, it identifies color and distance and checks the car’s position relative to the intersection and stop line.Source: Perception: Interpreting the World · p. 15

Reactive Behavior: Responding to the World

• The car’s reactive process predicts environmental responses, selects goals within road rules and physical limits, and plans actions that prioritize safety over comfort.At a stop sign, planning yields braking and scheduling selects a smooth deceleration to stop before the line.Source: Reactive Behavior: Responding to the… · pp. 15–16

Proactive Behavior: Staying Safe and Functional

• When an operating condition fails, the car generates recovery options and selects a permitted, feasible goal, such as recharging or returning to base.A battery level below 15% is one example of a condition that can trigger a meta-goal.Source: Proactive Behavior: Staying Safe and… · p. 16

2.4.2 How the Communicator Agent Works

• The Communicator interprets messages, checks them against knowledge-base constraints, and stores relevant information in its World and Self states.Semantic analysis includes identifying the sender, recipient, message type, content, and what the sender wants.Source: §2.4.2 · p. 16, Perception: Interpreting the World · pp. 16–17

Reactive Behavior: Responding to Messages

• For communication, the reactive process evaluates context and possible sender responses, then plans a norm-respecting action and prioritizes urgent messages.Possible actions include replying, forwarding, or requesting clarification; high-bandwidth tasks may be deferred when the network is congested.Source: Reactive Behavior: Responding to… · p. 17

Proactive Behavior: Staying Healthy

• The Communicator monitors its operating conditions and selects a feasible recovery goal, such as clearing messages when buffer space is low.If memory is unavailable, increasing buffer size is discarded as infeasible.Source: Proactive Behavior: Staying Healthy · p. 17

Our Contribution

• The architecture identifies the functional components every autonomous agent must possess and connects them through an agent knowledge base.It shares conceptual ground with BDI [10] but computes plans online rather than retrieving them from a predefined library.Source: §2.5 · pp. 17–18
• Unlike neurosymbolic approaches focused on reasoning and learning, the architecture integrates perception, decision-making, planning, and proactive self-maintenance into a framework for agency [13].Neurosymbolic AI combines neural-network perception with symbolic reasoning based on explicit knowledge [13].Source: §2.5 · p. 18
• Compared with world-model approaches that guide planning through predicted action outcomes [14], the architecture makes normative reasoning and proactive self-maintenance explicit components.The comparison concerns how goal selection and planning are shaped.Source: §2.5 · p. 18
• The architecture gives proactive self-maintenance a cognitive foundation for agency in physical and social environments, extending system-level resource-management frameworks [16].Those earlier frameworks emphasized self-configuration, self-healing, self-optimization, and self-protection.Source: §2.5 · p. 18
• The synthesis integrates perception with knowledge-guided reasoning, online planning, and proactive maintenance into one reference model for agents in dynamic environments with physical and social constraints.It also addresses gaps in agent-oriented engineering and context-aware multi-agent systems [18].Source: §2.5 · pp. 18–19

3. From Architecture to Implementation: Key Challenges

• A gap remains between the architecture’s principles for structuring and coordination and their actual implementation.The section turns from conceptual framework to questions that arise in creating concrete agents.Source: §3 · p. 19
• Implementation must link unstructured perception to structured knowledge, make safe predictions about a dynamic environment, and plan actions to achieve goals.Source: §3 · p. 19
• Integrating these solutions also raises systems-engineering complexity.Source: §3 · p. 19

• Perception must handle ambiguous or imprecise, high-volume sensory data and link extracted information to internal representations despite noise or varied wording.Source: §3.1 · p. 19
• The framing problem is deciding which information matters in an open, dynamic situation and which facts can be safely ignored.Source: §3.1 · p. 19
• RAG connects an LLM to long-term domain knowledge, which it retrieves before answering queries that exceed its general knowledge.Source: §3.1 · pp. 20–21, Figure 2 · p. 20
• Embedding retrieval scales and handles unstructured data but sacrifices precision and interpretability; graph retrieval supports reasoning but loses efficiency and scalability as graphs grow.Source: §3.1 · pp. 20–21
• A unified retrieval framework that selects relevant knowledge according to mission, goals, norms, and current state remains unresolved.Source: §3.1 · p. 21
• 0.9 per-query success yields 0.3487 overall reliability across 10 queries, illustrating compounding failures in unattended workflows.Source: §3.1 · p. 21

• The reactive module builds a predictive environment model from mission goals, relying on accurate state information and knowledge-base updates to supplement partial observations.Source: §3.2 · pp. 21–22
• Unlike structured-data automation, agents must construct semantic models of ambiguous real-world situations, sometimes combining perception with semantically rich maps.Source: §3.2 · p. 22
• New situations require agents to recognize gaps in their knowledge and adapt by discovering concepts or adding goals as circumstances change.Source: §3.2 · p. 22
• Inference can derive new knowledge from existing knowledge, but requires formal knowledge representation and management that current machine-learning AI does not support.Source: §3.2 · p. 22

• The predictive model alternates agent-controlled and environmental actions; exploring reachable states has exponential complexity and is suited to simple systems or limited horizons.Source: §3.3 · pp. 22–23
• Current machine-learning planners and chain/tree-of-thought methods do not react to changing environments during planning, limiting them to static or offline tasks.Source: §3.3 · pp. 22–23
• Planning in changing settings requires anticipating other agents' reactions and adapting over time, rather than treating plans as fixed action sequences.Source: §3.3 · p. 23
• A planner should generate predictive-model subtrees that preserve invariant goals under environmental actions while keeping transient goals achievable.Source: §3.3 · p. 23
• Standard reinforcement learning optimizes expected return but does not guarantee invariant safety constraints, and agents may violate constraints in unseen states.Source: §3.3 · p. 23
• World models simulate possible system evolutions from a state, but image analysis alone may not capture objects' dynamic properties.Source: §3.3 · pp. 23–24

3.4.1 Distributed Computation and Coordination

• A multi-agent system's effectiveness depends on coordinating its agents, and individually safe agents can still produce dangerous system-level deadlocks, livelocks, or emergent behavior.Source: §3.4.1 · p. 24
• Centralized coordination strains scalability, fault tolerance, and latency, while full decentralization complicates coordination and prediction; a hierarchical middle ground may manage complexity.Source: §3.4.1 · p. 24
• Distributed-systems methods could help with consensus and shared knowledge, but adapting them to unstructured agent messages remains needed and beyond this article's scope.Source: §3.4.1 · p. 25

3.4.2 Trust, Communication, and Institutional Organization

• Effective integration must align agents' communication and institutional relationships with their needs and roles, joining technical integration with functional and cognitive integration.Source: §3.4.2 · pp. 25–26

Communication

• Agents may communicate when they recognize a knowledge gap another agent can fill, but norms, incentives, and sanctions are needed to align individual interests with collective goals.Source: Communication · pp. 26–27
• Communication requires both channels and protocols and the ability to understand messages, intentions, and norms; trust mechanisms based on identity and capability indicators remain poorly understood.Source: Communication · p. 26

Institutional Organization

• Hierarchical authority can support command and control but risks higher-level failures, whereas distributed authority is more resilient but may be inefficient or conflict-prone.Source: Institutional Organization · pp. 27–28
• Institutional organization cannot be reduced to technical infrastructure: norms, roles, authority, and coordination must align for individual rule-following to produce coherent global organization.Source: Institutional Organization · pp. 27–28

4.1 Behavioral vs. Cognitive Properties: Why Behavior Is Not Enough

• Cognitive properties concern how an agent thinks and decides; their validity depends on its internal knowledge, reasoning, norms, and value systems, not merely its outputs.The authors connect normative and intent-directed properties to agents' use of knowledge when selecting actions and goals.Source: §4.1 · p. 28, §4.1.2 · p. 29

4.1.1 What the Agent Does: Behavioral Properties

• Behavioral properties concern observed input-output behavior and include safety, security, and usefulness, but they do not capture the full trustworthiness of agents designed to mimic cognition.Figure 3 classifies behavioral and cognitive properties using the distinction between risk-related and progress properties.Source: §4.1.1 · pp. 28–29

4.1.2 How the Agent Thinks: Cognitive Properties

• Trustworthy agents need both normative compliance and intent-directed deliberation: they must follow governing norms while making optimal, consistent, and competent choices.Normative properties concern conformity with rules; intent-directed properties concern how agents choose and plan goals, based on knowledge and reasoning.Source: §4.1.2 · pp. 29–30

4.1.3 Why Cognitive Properties Matter: Two Examples

• Human-driver and medical-exam examples show that passing behavioral tests does not establish the understanding, judgment, empathy, or accountability expected of a trustworthy agent.The authors distinguish controlled imitation from reliability across contexts and argue that multiple-choice exams cannot test the cognitive properties associated with medical practice.Source: §4.1.3 · p. 31

4.2 The Responsible AI Mirage: A Critical Assessment

• Responsible-AI frameworks are criticized for lacking implementation methods, while behavioral evaluation alone cannot establish intent, accountability, or understanding of norms.The authors argue that validating cognitive properties requires access to internal knowledge and reasoning, which they describe as extremely difficult with the current state of the art.Source: §4.2 · pp. 31–33

4.3 Specifying and Validating Trustworthiness

• Trustworthiness has behavioral and cognitive dimensions, but defining cognitive properties rigorously for current AI systems—and validating them—remains an open challenge.Behavioral properties pose a validation challenge; cognitive properties raise the more fundamental questions of definition and validation.Source: §4.3 · p. 33

4.3.1 Testing Behavioral Properties

• Neural-network properties are tested empirically, so testing can reveal defects but cannot prove their absence; AI testing also lacks meaningful coverage criteria for black-box systems.The authors contrast this with traditional-system verification, which can provide proof of a property under a model.Source: §4.3.1 · p. 33

4.3.1.1 Technical Systems: Testing Autonomous Vehicles

• Autonomous-vehicle testing needs scenario-selection criteria because the scenario space is vast and random simulation, even over many miles, cannot provide sufficient evidence.The criteria should target hazardous situations using statistical analysis of real-world conditions; static road layouts and dynamic obstacle scenarios both contribute to the testing complexity.Source: §4.3.1.1 · p. 34

4.3.1.2 Non-Technical Systems: The Interpretation Problem

• For LLMs, test results are subjective when input-output relationships require interpretation, and natural-language testing lacks clear coverage criteria, prompting reliance on expert benchmarks and human oracles.The authors distinguish fact-based checks, which may be unambiguous, from assessments such as bias or fairness.Source: §4.3.1.2 · p. 35

4.3.2 Cognitive Properties: The Ontological Gap

• Agent architectures exposing explicit knowledge and intentions could support cognitive-property specification through agent logics, but what it means to attribute responsibility to artificial agents remains unresolved.The authors identify access to knowledge bases and reasoning processes as relevant, while noting that responsible action also implies intention, accountability, norm understanding, and bearing consequences.Source: §4.3.2 · pp. 35–36

5 From Vision to Reality: The Path Forward

• Addressing autonomous systems’ perception, knowledge-representation, decision-making, and integration problems requires machine-learning and symbolic-AI fusion within an updated systems-engineering framework.Communication protocols provide channels for agents but do not resolve what they should communicate, why, or how to make interactions reliable and trustworthy.Source: §5 · p. 37
• More than 40% of agentic AI projects are predicted to be cancelled by the end of 2027 because of escalating costs, unclear business value, and inadequate risk controls.The authors also report that only about 130 of thousands of companies claiming agentic capabilities are building systems that genuinely merit the label; rebranding chatbots as agents without meaningful outcomes is called “agent washing” [51].Source: §5 · p. 37
• Roughly three-quarters of enterprises were adopting agentic AI, but only a tiny minority were running it in production, exposing a capability-deployment verification gap.The gap concerns verifying or trusting an agent on proprietary systems and live data after it performs a task in a controlled test.Source: §5 · p. 37
• The study’s main result is a reference architecture whose knowledge attributes shape diverse cognitive behaviors and which can be specialized for different missions and environments.Implementing agents based on the architecture still poses challenges.Source: §5 · p. 37
• Formal validation is impossible given autonomous systems’ reactive and architectural complexity and multi-agent dynamism, and undesirable emergent properties may persist even when individual agents are safe.In one reported case, autonomous vehicles’ separate route optimizations contributed to a traffic jam that no single vehicle could resolve.Source: §5 · pp. 37–38
• Passing a benchmark demonstrates skills only under controlled conditions, while cognitive-property validation requires access to internal knowledge and reasoning processes that current AI systems do not provide.The authors identify agent logics as a framework for specifying properties such as rationality and normative compliance, but say practical validation methods are still needed.Source: §5 · p. 38
• The authors argue that systems whose required cognitive properties cannot be verified should not be deployed where trust is essential, and that accountability cannot be assigned to machines as it is to humans.They state that transparency, responsibility, and ethical values must be built in from the start, rather than added to an opaque system.Source: §5 · p. 38