Source-linked AI summary
Cognitive Architectures for Language Agents
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, Thomas L. Griffiths
TL;DR
Language agents have achieved empirical success, but lack a systematic framework for organizing existing agents and planning future developments. The paper proposes CoALA, which structures agents around memory, actions, and decision-making, uses it to organize recent work, and identifies directions toward more capable agents.
Problem
Existing language agents show empirical success, but the field lacks a systematic framework to organize them and plan future developments.
Method
CoALA is a conceptual framework that organizes language agents by modular memory, internal and external actions, and an interactive decision-making loop with planning and execution.
Results
CoALA expresses a large body of existing agents and identifies underexplored directions for developing new ones.
Takeaways & Limitations
The framework contextualizes current language agents within AI's broader history and outlines a path toward language-based general intelligence.
Takeaways & Limitations
Learning new procedures for learning or decision-making is theoretically possible but risky, and no language agents currently implement it.
Abstract
from arXiv · showhide
Recent efforts have augmented large language models (LLMs) with external resources (e.g., the Internet) or internal control flows (e.g., prompt chaining) for tasks requiring grounding or reasoning, leading to a new class of language agents. While these agents have achieved substantial empirical success, we lack a systematic framework to organize existing agents and plan future developments. In this paper, we draw on the rich history of cognitive science and symbolic artificial intelligence to propose Cognitive Architectures for Language Agents (CoALA). CoALA describes a language agent with modular memory components, a structured action space to interact with internal memory and external environments, and a generalized decision-making process to choose actions. We use CoALA to retrospectively survey and organize a large body of recent work, and prospectively identify actionable directions towards more capable agents. Taken together, CoALA contextualizes today's language agents within the broader history of AI and outlines a path towards language-based general intelligence.
1 Introduction
Language agents extend LLMs beyond text generation by connecting them to environments, memory, and internal reasoning, but inconsistent terminology makes the field difficult to organize. CoALA addresses this gap with a framework spanning memory, actions, and decision-making.
- Language agents connect LLMs to internal memory and external environments, grounding limited knowledge and reasoning in stored information or observations.
- Recent cognitive language agents use LLMs for direct action selection, reasoning, planning, and long-term memory management.
- Custom terminology such as “tool use,” “grounding,” and “actions” makes agents difficult to compare, track over time, and develop with consistent abstractions.
- CoALA organizes general-purpose language agents by information storage, action space, and an interactive decision-making loop with planning and execution.
- CoALA combines a theoretical framework with a survey of empirical work, enabling identification of short-term and long-term directions for future agents.
2 Background: From Strings to Symbolic AGI
Production systems evolved from symbolic rewriting rules into controlled procedures for complex behavior, while cognitive architectures added memory, perception, grounding, and decision-making. These ideas provide historical context for language agents, whose flexible text-based representations and learned production distributions may address some limitations of traditional symbolic systems.
- 2.2 Control flow: From strings to algorithms: Production systems apply rules iteratively to generate outcomes, and control flow turns them into algorithms by determining which productions execute.
- 2.2 Control flow: From strings to algorithms: Markov algorithms use priority ordering to select productions, and simple production rules can produce Turing-complete behavior.
- 2.3 Cognitive architectures: From algorithms to agents: Cognitive architectures extend production systems with processes such as perception, memory, planning, sensory grounding, and action selection.
- 2.3 Cognitive architectures: From algorithms to agents: Soar stores productions in long-term memory, matches them against working memory, ranks candidate actions, and implements the selected internal or external action.
- 2.3 Cognitive architectures: From algorithms to agents: Soar supports storing facts and experiences in long-term memory and modifying behaviors through reinforcement learning.
- 2.3 Cognitive architectures: From algorithms to agents: Traditional cognitive architectures are constrained by logical-predicate domains and the need for many pre-specified rules.
- 2.4 Language models and agents: LLMs operate over arbitrary text and learn distributions over productions through pre-training, potentially making cognitive architectures more flexible than logic-based systems.
- 2.4 Language models and agents: Language models generate text from context by sampling tokens from a learned conditional distribution.
3 Connections between Language Models and Production Systems
The paper draws a natural analogy between production systems and language models because both operate through string expansions or modifications. This analogy extends to prompting, language agents, and cognitive architectures as increasingly structured control processes.
- Production systems and language models both define possible expansions or modifications of strings, motivating their conceptual correspondence.Production systems rewrite strings through operations, while language models generate continuations of prompts.
- LLMs can be viewed as probabilistic production systems that assign P(Y_i|X) to possible completions given input X.Each model call samples a possible completion according to the distribution induced by the model.
- LLM opacity and probabilistic randomness make language-model behavior harder to analyze or control than discrete, human-legible production rules.Traditional production systems expose their rules, whereas LLMs rely on billions of uninterpretable parameters.
- Prompt engineering as control flow: Prompt engineering and chaining define sequences of productions by preprocessing inputs, eliciting reasoning, and combining multiple LLM calls.These methods range from fixed manipulations to dynamic prompts and increasingly complicated algorithms.
- Towards cognitive language agents: Language agents extend these control processes from predefined prompt chains to interactive environmental feedback, intermediate reasoning, and experience-based adaptation.Recent agents use reasoning, planning, long-term memory, and procedural knowledge to guide action selection and future behavior.
4 Cognitive Architectures for Language Agents (CoALA): A Conceptual Framework
CoALA frames language agents through modular memory, internal and external actions, and an interactive planning-and-execution loop. It extends cognitive-architecture ideas with LLM-based reasoning and identifies learning and retrieval capabilities that remain underdeveloped.
- Framework overview: CoALA organizes language agents by working and long-term memory, internal and external actions, and interactive decision-making.The framework positions the LLM within a larger cognitive architecture.
- Action space: External actions ground agents in environments, while internal actions retrieve, reason over, or learn from memory.Internal actions differ by the accessed memory and whether information is read or written.
- Decision-making: Each decision cycle uses reasoning and retrieval for planning before executing grounding or learning actions.The cycle continuously accepts perceptual input and calls action procedures in response.
- Learning and novelty: LLM-based reasoning produces flexible knowledge and heuristics, replacing hand-written rules while text serves as the default internal representation.The framework also highlights code updates, parameter updates, retrieval updates, and memory modification as learning possibilities.
- Memory: CoALA distinguishes episodic, semantic, and procedural long-term memory for experiences, world knowledge, and procedures.Semantic memory may be expanded through LLM reasoning, while procedural memory includes LLM weights and agent code.
- Memory: Working memory persists across LLM calls and connects perceptual inputs, generated or retrieved knowledge, long-term memories, and grounding interfaces.LLM inputs are synthesized from working-memory contents, and outputs are parsed back into stored variables.
- Open directions: Context-specific recall remains understudied, and updating learning or decision-making procedures is both risky and not currently implemented by known language agents.The paper also identifies modifying and deleting memory as understudied.
- Learning and novelty: Language agents can store task-relevant language and select diverse learning procedures, potentially enabling rapid learning and compounded self-improvement.The paper contrasts this with parameter updates, which are described as more costly.
5 Case Studies
The case studies cast diverse language agents into CoALA’s memory, action, and decision-making dimensions. They show that the framework captures both minimal agents and systems combining all four action types with long-term procedural learning.
- Cross-agent comparison: CoALA characterizes agents across domains ranging from Minecraft and robotics to pure reasoning and social simulation.The framework exposes similarities and differences through a common structure.
- SayCan: SayCan uses only external grounding skills and procedural memory, without internal reasoning, retrieval, or learning actions.Its fixed action space contains 551 grounding skills for robotic kitchen interactions.
- ReAct: ReAct combines reasoning and external grounding but lacks semantic and episodic memory, retrieval, and learning actions.Its fixed cycle uses reasoning to analyze situations and remake plans before generating a grounding action.
- Voyager: Voyager combines grounding, reasoning, retrieval, and learning through a hierarchical library of code-based Minecraft skills.Successful procedures are written to procedural memory, while failures trigger refinement and re-execution.
- Voyager: Voyager outperforms ReAct, AutoGPT, and procedural-memory ablations on exploration, technology-tree mastery, and zero-shot generalization to unseen tasks.The passage attributes these comparisons to empirical verification of long-term memory and procedural learning.
- Generative Agents: Generative Agents use retrieval and reasoning over episodic memories to generate reflections that are written into semantic memory.They are grounded in a sandbox game and interact with the environment and other agents.
- Tree of Thoughts: Tree of Thoughts has one external action, no long-term memory, and only reasoning internally, but uses deliberate proposal, evaluation, and selection during planning.Its external action is submitting a final solution to a reasoning problem.
6 Actionable Insights
CoALA provides a cognitive-architecture-based framework for organizing language agents and proposes modular, structured directions for improving their reasoning, memory, actions, and decision-making.
- CoALA complements empirical language-agent surveys with a theoretical framework grounded in cognitive architectures.
- Modular agents: thinking beyond monoliths: Modular agent structures could consolidate technical investment, improve compatibility, and support component reuse across deployments.Standardized terms and open-source implementations would facilitate plug-and-play modularity; industry libraries could also reduce technical debt.
- Structured reasoning: thinking beyond prompt engineering: Structured reasoning should update working-memory variables through explicit procedures rather than relying only on low-level prompt manipulation.Agent reasoning use cases such as self-evaluation, reflection, and action generation could also inform future LLM training.
- Long-term memory: thinking beyond retrieval augmentation: Memory-augmented agents can autonomously read and write self-generated content, supporting lifelong learning from knowledge, experience, and skills.Agents could combine human-written semantic knowledge with self-generated episodic and procedural knowledge, while integrating retrieval with forward simulation to ground planning.
- Action space: thinking beyond external tools or actions: CoALA defines action spaces with internal reasoning, retrieval, and learning actions alongside external grounding actions to systematize agent design.Larger action spaces increase decision-making complexity, while learning and grounding actions can create internal or external safety risks.
- Future agents should extend deliberative reasoning to grounded, long-term tasks and adapt computation to the utility of planning.The paper also identifies mixed language-and-code planning, metareasoning, calibration, and alignment as important directions.
7 Discussion
The discussion examines how agent design may change with model capabilities, modalities, and environment boundaries. It argues that CoALA’s architectural principles remain useful while highlighting trade-offs in integration, controllability, and autonomous learning.
- LLMs vs VLMs: should reasoning be language-only or multimodal?: Integrated multimodal agents bypass lossy image-to-text conversion but couple perception and reasoning, making them more domain-specific and harder to update.Modular systems translate perceptual data into language, whereas integrated systems project images into the language model’s representation space.
- Internal vs. external: what is the boundary between an agent and its environment?: The internal–external boundary can be determined by controllability and coupling rather than by whether a resource or operation appears digital.An agent-only writable offline database can count as internal memory, while execution on an external machine counts as external grounding.
- Physical vs. digital: what differences beget attention?: Digital environments permit sequential resets and parallel trials, potentially enabling decision procedures unlike those used by biological agents.
- Learning vs. acting: how should agents continuously and autonomously learn?: CoALA treats learning as a deliberate result action that agents choose alongside external actions, rather than as a fixed schedule separate from decision-making.This design lets agents defer learning until an appropriate time.
- GPT-4 vs GPT-N: how would agent design change with more powerful LLMs?: More powerful LLMs could change the relative importance of memory, grounding, learning, and planning components in agent design.Longer context could reduce the importance of long-term memory, while stronger reasoning could support longer-horizon planning.
8 Conclusion
The paper proposes CoALA as a conceptual framework for describing and building language agents. Drawing on symbolic AI and cognitive science, it connects historical insights to frontier LLM research and suggests a path toward more general, human-like AI.
- CoALA provides a conceptual framework for describing and building language agents.
- The framework draws on symbolic artificial intelligence and cognitive science to connect historical insights with frontier LLM research.
- CoALA is presented as a path toward more general and more human-like artificial intelligence.