Source-linked AI summary
CEAA: A Cognitive Embodied Agents Architecture for Interactive Computing Systems
Aimilios Hadjiliasi, Louis Nisiotis
TL;DR
Embodied IVAs need cognitive architectures that are practical to implement in real-time 3D environments, but existing approaches often separate cognitive depth from embodied execution. This paper proposes CEAA, a modular framework, and demonstrates its feasibility through prototype desktop and VR learning environments with embodied pedagogical agents.
Problem
Existing cognitive architectures often lack implementation guidance and real-time feasibility, while game-AI approaches can prioritize embodiment over cognitive depth and explainability.
Method
CEAA integrates Sense-Think-Act, shared knowledge, explicit memory, reasoning, planning, and embodied actions within a modular architecture for real-time IVA deployment.
Results
CEAA feasibility was demonstrated in Unity3D desktop and VR museum environments featuring four embodied pedagogical agents with distinct educational roles.
Takeaways & Limitations
CEAA provides an implementation-oriented reference for connecting cognitive functions, runtime decisions, and embodied behavior in interactive virtual environments.
Takeaways & Limitations
CEAA remains conceptual and theoretical, and requires benchmarking against other architectures across latency, development effort, and user experience.
Abstract
from arXiv · showhide
The development of embodied Intelligent Virtual Agents (IVAs) that have cognitive capabilities in real-time interactive virtual environments remains a challenge, even with today's advancements in technology. Existing architectures are often focused on either the implementation of low-level reactive control systems that are constrained by commercial game engines, or high-level representations of reasoning models that can be difficult to implement in virtual worlds. This paper builds on that notion and proposes a modular cognitive architecture for deploying embodied IVAs. This architecture builds on existing, pre-established frameworks such as the Sense-Think-Act paradigm and the Belief-Desire-Intention cognitive model, among others, and aims to provide a reusable implementation-oriented framework as a template for deploying IVA "brains" in interactive 3D computing systems. The proposed architecture contributes by providing a modular, implementation-oriented framework for the deployment of embodied, cognitive-capable IVAs and bridges the gap between high-level agent reasoning models with real-time embodied execution, for scalable, adaptive, and explainable agents in complex interactive virtual environments.
I. INTRODUCTION
The introduction frames embodied IVAs as agents that must operate autonomously, interact naturally, and adapt over time in virtual environments. It identifies a gap between reactive implementations and cognitively expressive, real-time deployable architectures, which CEAA addresses through modular separation of cognition and embodied action.
- Embodied IVAs in virtual environments are expected to operate autonomously, interact naturally with users, and adapt their behavior over time.
- The introduction identifies a lack of implementation-oriented cognitive architectures unifying high-level reasoning, memory, planning, and real-time embodied action in dynamic interactive environments.
- CEAA is proposed as a backbone for deploying the brains of multiple cognitively capable IVAs through a clear separation of cognitive states, reasoning, planning, and embodied actions.
- This separation facilitates integration with modern game engines such as Unity and Unreal.
II. BACKGROUND AND CONTEXT · A. Intelligent Virtual Agents and Embodiment in Virtual Worlds
Intelligent Virtual Agents are autonomous or semi-autonomous AI software entities that interact with users in virtual environments while perceiving, reasoning, and acting within situated worlds. Embodiment gives these agents a visible, typically human-like presence but introduces requirements beyond classical agent intelligence, including decision-making, suggestions, and assistance.
- A. Intelligent Virtual Agents and Embodiment in Virtual Worlds: IVAs have been extensively investigated for more than two decades as autonomous or semi-autonomous entities in virtual environments.The passage characterizes them as a longstanding research subject.
- A. Intelligent Virtual Agents and Embodiment in Virtual Worlds: IVAs are advanced AI-powered software designed for natural, conventional, and personalized interactions with users in digital systems.Their interaction capabilities are described as both conventional and personalized.
- A. Intelligent Virtual Agents and Embodiment in Virtual Worlds: Situated IVAs can perceive, reason, and act in the virtual worlds where they operate.These capabilities connect the agents’ intelligence to their virtual environment.
- A. Intelligent Virtual Agents and Embodiment in Virtual Worlds: Embodiment introduces drawbacks for IVAs in immersive and interactive virtual environments.The passage presents embodiment as adding challenges rather than only benefits.
- A. Intelligent Virtual Agents and Embodiment in Virtual Worlds: Embodied agents face requirements beyond classical agent intelligence, including decision-making, suggestions, and assistance.These additional expectations arise in immersive and interactive environments.
- A. Intelligent Virtual Agents and Embodiment in Virtual Worlds: Embodiment gives an agent a visible presence in its environment, typically through a human-like representation.The passage notes that embodiment may take any form, although human-like representations are typical.
B. Agent Architectures and Cognitive Models
Agent architectures provide the structural foundation for intelligent behavior and can be analyzed across execution, behavioral, and cognitive levels of abstraction. Key paradigms and models include Sense-Think-Act, BDI, SOAR, and ACT-R, each addressing different aspects of embodied agent intelligence.
- Levels of Abstraction: Agent architectures form the structural foundation for implementing intelligent behavior and span execution, behavioral, and cognitive abstraction levels.The execution level defines when computation occurs, while behavioral mechanisms organize agent behavior.
- Sense-Think-Act: Sense-Think-Act models agents as systems that perceive their environment, think internally, and execute actions.The paradigm clarifies internal processes but may not specify complex reasoning and memory mechanics sufficiently.
- BDI Architecture: BDI provides a higher-level framework for autonomous agents using beliefs, desires, and intentions to represent practical reasoning.It was inspired by Michael Bratman’s philosophical model of human practical reasoning.
- Cognitive Architectures: SOAR and ACT-R model cognition through production rules, memory subsystems, and learning mechanisms for human intelligence, problem-solving, and cognition.SOAR uses a problem-space model for perception, planning, and learning, whereas ACT-R structures declarative and procedural memory for goal management.
C. Knowledge Representation and Memory in Virtual Agents
Memory and knowledge representation are foundational to agents because they determine how information is encoded, stored, and used over time. Classical cognitive architectures typically represent knowledge symbolically through facts and rules distributed across distinct memory subsystems, including episodic and procedural memory.
- C. Knowledge Representation and Memory in Virtual Agents: Memory and knowledge representation determine how agents encode, store, and exploit information over time.These components are identified as key parts of any type of agent.
- C. Knowledge Representation and Memory in Virtual Agents: Classical cognitive agent architectures typically represent knowledge symbolically using stored facts and rules.The symbolic representation is organized in distinct memory sub-systems.
- C. Knowledge Representation and Memory in Virtual Agents: Episodic and procedural memories store the short- and long-term facts agents need to know.These memory types are presented as examples within classical cognitive agent architectures.
D. Embodied IVAs in Commercial Game Engines
Commercial game engines such as Unity and Unreal Engine constrain IVA deployment through tightly integrated, component-based execution models. Although they support embodied agents and reactive behavior, they provide limited support for high-level cognition.
- Engine Constraints: Unity and Unreal Engine constrain IVA deployment through their development capabilities and execution models.These engines are designed primarily for games, simulations, and virtual environments.
- Engine Constraints: Component-based, object-oriented, and event-driven execution tightly integrates environment, agent, physics, animation, and rendering logic.This integration spans the core systems involved in developing interactive virtual environments and agents.
- Engine Constraints: Despite supporting agent embodiment, navigation, behavior, and interaction, current engines offer limited high-level cognition beyond reactive approaches.The engines provide solid development tools but do not adequately support cognition-oriented agent development.
III. PROPOSED SYSTEM ARCHITECTURE
CEAA is a reusable, implementation-oriented architecture for embodied agents that combines the sense–think–act paradigm with BDI-based cognitive reasoning. Its three layers and twelve interconnected components support perception, knowledge management, reasoning, planning, and action in virtual environments.
- Architecture overview: CEAA provides a reusable template for implementing the “brain” of individual embodied agents in virtual environments.It extends sense–think–act with BDI reasoning and adds components for shared knowledge representation and memory management.
- Architecture overview: The architecture is separated into three layers and twelve interconnected components that collectively support perception, reasoning, and action.The layers comprise the User and Environment Layer, Knowledge Layer, and agent reasoning and action functions.
- Knowledge and perception: The Knowledge Base maintains environmental events and world state as a centralized shared blackboard used collaboratively by multiple agents and specialized knowledge sources.It records changes originating from users, agents, virtual objects, and system processes in a structured form.
- Knowledge and perception: The Sense and Memory components select relevant information and organize past experiences, semantic knowledge, and user models to support continuity and context-aware behavior.Relevant events identified through the blackboard are forwarded to Memory for storage.
- Cognitive reasoning and action: Think orchestrates Memory, Reasoner, and Planner interactions, while the BDI-based Cognitive Construct and Reasoner support adaptive, explainable, goal-oriented decision-making.The Planner transforms beliefs, desires, and intentions into structured action sequences, and Act interfaces cognition with the environment.
IV. PROTOTYPE-BASED FEASIBILITY EVIDENCE
CEAA was implemented in parallel desktop and VR virtual-museum environments with embodied pedagogical agents supporting instruction, evaluation, and learner interaction. A comparative study found positive technology acceptance in both conditions, while further evaluation is needed for performance, scalability, and generalizability.
- Prototype implementation: CEAA informed two identical Unity3D learning environments, one for desktop and one for VR, modeling a virtual museum dedicated to ENIAC and early computing history.The museum included learning materials, an interactive three-dimensional ENIAC representation, and four embodied pedagogical agents with different responsibilities.
- Prototype implementation: The agents acted as navigators, instructors, and evaluators, delivering ENIAC presentations, answering questions, conducting Q&A assessments, and monitoring learners and the environment.Their embodied capabilities included speech, gaze, gestures, facial expressions, movement, spatial awareness, navigation, and obstacle avoidance.
- Evaluation: Technology acceptance was positive in both conditions, with participants rating both systems relatively easy to use and useful for learning and reporting positive attitudes and future-use intentions.VR showed slightly higher median effort expectancy and behavioral intention, while performance expectancy and attitude towards use had similar median values across conditions.
- Limitations: CEAA supported deployable multi-agent VR and desktop experiences, but further technical evaluation is required to establish performance, scalability, and generalizability across application domains.The limitation concerns the architecture’s broader technical validation beyond the demonstrated prototype.
V. DISCUSSION
The proposed architecture addresses practical challenges in deploying cognitively capable agents in real-time virtual environments. Its feasibility was demonstrated in prototype learning environments populated by pedagogical agents with different educational roles.
- Practical deployment: The architecture responds to practical challenges in deploying agents with cognitive capabilities in real-time virtual environments.This frames the work as addressing deployment feasibility rather than only theoretical modeling.
- Prototype validation: Its feasibility was demonstrated through prototype learning environments populated by pedagogical agents.The prototypes served as environments for demonstrating the architecture’s feasibility.
- Prototype validation: The prototype environments included pedagogical agents with different educational roles.The agents were instantiated with varied educational responsibilities.
A. Development and Implementation Considerations
CEAA’s real-time deployment depends on separating frame-critical functions from expensive cognitive operations. Its modular components can interface with engine systems or run independently, making asynchronous execution central to feasibility.
- Latency Management: Frame-critical sensing, behaviour triggering, animation control, and action execution should remain lightweight and near the environment’s real-time update loop.Computationally expensive cognitive operations should be separated from these latency-sensitive processes.
- Modular Deployment: Environment and Act components can interface with scene objects, physics, animation, and interaction systems, while cognitive components operate as independent services or manager components.The Knowledge Base, Memory, Reasoner, and Planner are identified as independently deployable components.
- Modular Deployment: The Behaviour Mapper translates abstract decisions into engine-specific animations, dialogue, navigation commands, or other interaction scripts.This translation connects high-level decisions with concrete interactive execution.
- Deployment Feasibility: CEAA is a deployment template rather than a computationally fixed implementation, and its real-time feasibility depends on asynchronous execution.The architecture therefore permits different deployment arrangements for its modular components.
B. Contributions
CEAA contributes an implementation-oriented framework that translates established cognitive concepts into modular components for embodied IVAs in real-time virtual environments. It integrates cognition and embodiment within a unified decision-making process while extending beyond standalone BDI or blackboard architectures.
- Primary contribution: CEAA provides an implementation-oriented architectural framework for cognitive, goal-directed, adaptive, and embodied IVAs in real-time virtual environments.It translates established theoretical concepts into modular components implementable within interactive 3D systems.
- Cognition and embodiment: CEAA gives cognition and embodiment equal importance by embedding behavior expression within the decision-making pipeline.This integration targets expressive and contextually appropriate behavior generation in IVAs.
- Architectural integration: CEAA links beliefs, desires, and intentions with shared environmental knowledge, agent-specific memory, capability-aware planning, and embodied behavior execution.It uses a blackboard as one shared-knowledge layer rather than as the complete control mechanism.
VI. CONCLUSIONS, LIMITATIONS AND FUTURE WORK
CEAA is proposed as a modular, implementation-oriented architecture for deploying embodied IVAs with cognitive functions in real-time virtual environments. Although prior studies tested its feasibility, the architecture remains conceptual and theoretical, motivating future benchmarking against other architectures and components.
- Conclusions: CEAA provides a modular, implementation-oriented architecture for deploying embodied IVAs with cognitive function in real-time virtual environments.It was developed to address the disconnection between cognitively capable agent deployment and practical development in real-time interactive 3D systems.
- Conclusions: The architecture builds on several existing architectures and paradigms to support cognitively capable embodied agents.
- Limitations: Although CEAA’s feasibility was tested in prior studies, it remains at a conceptual and theoretical level.This conceptual status is identified as the architecture’s most significant limitation.
- Future Work: Future studies will benchmark CEAA and compare its capabilities with other architectures and components, including BDI and sense-think-act.The planned comparisons are intended to examine performance across multiple dimensions.