Projective simulation for artificial intelligence

Hans J. Briegel; Gemma De las Cuevas

Source-linked AI summary

Projective simulation for artificial intelligence

Hans J. Briegel, Gemma De las Cuevas

arXiv:1104.3787v3Apr 19, 2011nlin.AOcond-mat.dis-nnquant-ph

TL;DR

The paper addresses how an embodied learning agent can use simulation to project possible future situations before acting. It proposes projective simulation as a random walk through a dynamically changing episodic-memory network, illustrates learning behavior in an invasion game, and extends the framework toward quantum operation.

Problem
The paper seeks a physically grounded and sufficiently general simulation concept for embodied agents that can support learning and intelligent action.
Method
The model uses a projection simulator that performs random walks through episodic-memory clips, recombines experience, evaluates simulated sequences, and updates memory through learning.
Results
The authors report that projective simulation enhances learning capabilities, supports elementary creative action, and was tested with a programmed agent in an invasion game.
Takeaways & Limitations
Projective simulation provides a general framework connecting embodied artificial-agent design with reinforcement learning, physics, and quantum computation.

Abstract

from arXiv · show

We propose a model of a learning agent whose interaction with the environment is governed by a simulation-based projection, which allows the agent to project itself into future situations before it takes real action. Projective simulation is based on a random walk through a network of clips, which are elementary patches of episodic memory. The network of clips changes dynamically, both due to new perceptual input and due to certain compositional principles of the simulation process. During simulation, the clips are screened for specific features which trigger factual action of the agent. The scheme is different from other, computational, notions of simulation, and it provides a new element in an embodied cognitive science approach to intelligent action and learning. Our model provides a natural route for generalization to quantum-mechanical operation and connects the fields of reinforcement learning and quantum computation.

I. INTRODUCTION

The paper proposes projective simulation as an embodied learning framework in which agents use episodic memory to simulate possible future situations before acting. It distinguishes this experience-based projection from dynamic physical simulators and connects agent design with physics and quantum computation.

Intelligent agents must perceive and act autonomously in complex, changing environments, motivating learning-oriented models of behavior.
The projection simulator lets an agent use previous experience and variations of it to project itself into potential future situations before real action.
Episodic and compositional memory represents experience as a stochastic network of clips that can be replayed and recombined into fictitious sequences.
Projective simulation is experience-based and differs from physical simulators that reproduce the dynamics of other systems.
During projection, fictitious memory sequences are evaluated and screened for features that can lead to concrete action.
The framework is intended to connect embodied agent research with physics and support generalization to quantum simulation.

II. INTELLIGENT AGENTS

The paper adopts an embodied view of intelligent agents, treating perception and action as physical interaction mediated by sensors and actuators. It proposes dynamic episodic memory and projective simulation as mechanisms for creative behavior.

An agent perceives its environment through sensors and acts on it through actuators, with an internal computing device processing perceptual input.
The paper focuses on stochastic agents, for which percept histories determine probabilities over possible next actuator moves.
Embodied agents acquire information through physical interactions with their environments rather than through detached software design alone.
The proposed memory is a dynamic network of clips that grows as the agent interacts with the world and supports a defined notion of creative behavior.

III. LEARNING BASED ON PROJECTIVE SIMULATION

Projective simulation gives a learning agent access to both remembered and self-generated fictitious experience before it chooses an action. Episodic memory is modeled as a probabilistic clip network that can change through learning and composition.

The agent’s decisions depend on previous experience and on fictitious experience generated by its own projection simulator and episodic memory.
Perceptual input triggers a random walk through episodic memory, whose recalled content is evaluated before producing concrete action.
Memory changes through actual experience and compositional recall, allowing new content corresponding to experience that never occurred.
Episodes or clips are stored patches of previous experience that function as variable units the agent can access, manipulate, and create.
Episodic memory is a probabilistic clip network in which excited clips call neighboring clips with probabilities influenced by percept history.
The simple examples illustrate projective simulation in reinforcement-learning problems while presenting it as a more general building block for agent architectures.

IV. MATHEMATICAL MODELING AND NOTATION

The mathematical model represents an agent as a stochastic mapping from percepts to actions and specifies how percepts trigger memory excitation, clip transitions, and action activation. Its memory consists of finite, probabilistic networks of remembered percepts and actions.

The agent’s policy is the conditional probability P^(t)(a|s) of action a given percept s at time t.
Memory is a network of episodes or clips containing sequences of remembered percepts and actions.
An operation cycle encounters a percept, excites an initial clip through an input-coupler probability, performs a random walk through clips, and exits memory by activating an action.
The formal setting restricts percepts, actions, and clips to finite-sized sets for finite agents acting in finite worlds.
Percept and actuator spaces are Cartesian products whose components represent categories or degrees of freedom of inputs and actions.
Clips are sequences of remembered real or fictitious percepts and actions, with clip length L determining the sequence size.
Projective simulation uses memory-space random walks to replay and compose experience, while learning modifies transition probabilities through evaluation such as reinforcement learning.

V. SIMPLE EXAMPLE: INVASION GAME

The invasion game illustrates how a defender can learn to block an attacker by associating shown symbols with subsequent moves and improving its blocking efficiency through projective simulation. The agent represents percepts, actions, and feedback as clips and emotion-tagged transitions whose probabilities are updated from rewards.

The game evaluates learning by the defender’s blocking efficiency, namely how many attacker moves it can withstand before invasion.After an invasion, the duel ends and a new attacker appears at the defender’s current door.
The defender must infer whether the attacker will move left or right from a symbol shown before each move, then block the predicted door.A correct guess lets the faster defender arrive before the attacker; an unblocked door counts as a successful invasion.
The attacker’s symbols have unknown meanings, but their meanings remain stable long enough for the robot to learn the associated strategy.Symbols may be arrows, signed numbers, bodily directions, facial expressions, or abstract signals, provided the robot can distinguish them.
Projective simulation improves blocking efficiency by testing possible action sequences in episodic memory before selecting real movement, with simple reinforcement and associative learning as special cases.The example considers progressively more sophisticated simulation processes.
The memory represents symbols, defender movements, and emotional feedback as clips and transitions, with rewarded transitions changing emotion tags and influencing later simulation probabilities.For the simplest reward function, rewarded transitions receive feedback that amplifies their transition probability, while unrewarded transitions are attenuated or not amplified.

A. Projective simulation & learning without composition

The agent learns percept-action mappings by updating clip-transition weights from rewards and using simulation with reflection before real action. Learning improves with reflection and scales with environment size, while dissipation trades peak performance for adaptation speed.

Projective simulation: Perceptual input excites a percept clip, which stochastically transitions to an actuator clip that is translated into the corresponding real action.With reflection time R, emotion tags can mediate whether simulated transitions are coupled out into motor action.
Learning rules: Reward-dependent updates modify h-matrix weights and thereby change future transition probabilities for percept-action sequences.Positive feedback increases the probability of simulating and typically executing rewarded actions; unrewarded sequences decrease weights.
Adaptation and forgetting: Dissipation reduces maximum blocking efficiency by causing forgetting, but can shorten adaptation when previously learned symbol meanings are inverted.After the inversion at n = 250, weak dissipation requires longer unlearning, producing a trade-off between adaptation speed and achievable blocking efficiency.
Adaptation and forgetting: When new colors enlarge the percept space, the agent learns their meanings from scratch without the increased adaptation period caused by strategy inversion.The extended network does not exploit similarity between same-shaped symbols with different colors.
Reflection: Larger reflection times increase learning speed because virtual percept-action sequences are recalled with emotion tags before motor action.Iterated simulation makes the agent more likely to find the right actuator move before acting, provided simulation time does not become too long.
Scaling: Learning time increases linearly in both |S| and |A|, and reflection in clip space speeds learning.With N = |A| = |S|, the scaling is quadratic in N, matching the scaling observed for Q-learning and AHC in this game.

B. Projective simulation & learning with composition I

Projective simulation extends episodic memory with compositional clip sequences that let the agent test fictitious actions before acting. New percepts and associative links can accelerate transfer learning, but excessive deliberation or compositional strength can reduce performance.

Simulation process: The model generates fictitious clip sequences between percepts and actuators, screening them before translating a rewarded simulated action into real action.Deliberation length D controls the number of intermediate memory clips; D = 0 is direct percept-to-actuator action.
Network dynamics: Transition probabilities are determined by directed clip-network weights, while rewarded simulated transitions are strengthened after each cycle.The update rules also distinguish direct from compositional connections and include damping of network weights.
Network growth: The clip network can grow when a percept appears for the first time, making that percept accessible from other clips through initialized incoming connections.The incoming weights are initialized to K in the described rule, while outgoing-link activation can cause broader exploration and slower learning.
Associative learning: Associative links between percepts with equal shape but different colors let training on red symbols accelerate learning when blue symbols appear.The modified memory architecture uses compositional connections whose effectiveness increases with K in the reported comparison.
Trade-offs: Larger compositional strength K increases deliberation and can expose unrewarded transitions, reducing blocking efficiency or causing simulations to loop among similar percept clips.With a maximum deliberation time Dmax, performance for large K drops significantly because compositional links can dominate direct links.

C. Projective simulation & learning with composition II

Projective simulation can generate fictitious episodes by varying or merging remembered clips, enabling composite actions that were not previously in the agent’s active repertoire. In the two-dimensional invasion-game example, frequent activation of compatible actuator clips creates a merged clip that can be reinforced through experience.

Fictitious episodes: Fictitious episodes can be created during simulation by varying or merging stored episodes, after which real-world rewards select successful projections.The corresponding memory connections are enhanced when projected episodes lead to rewarded factual actions.
Network evolution: Frequent excitation of two actuator clips from one percept clip produces a novel merged clip that becomes part of the existing clip network.Figure 16 schematically illustrates this network evolution behind the learning curve in Figure 17.
Two-dimensional task: In the two-dimensional game, the agent initially moves only horizontally or vertically because diagonal motion is absent from its immediate action repertoire.The enlarged actuator space represents independent x- and y-direction motors, while diagonal movement requires simultaneous activation.
Composite action: Random clip composition can create a merged clip that triggers both motors, allowing the agent to simulate diagonal movement before attempting it in reality.This mechanism is especially relevant when multiple actions receive rewards or when moves are ambivalent.
Composite action: A merging threshold can require sufficiently strong connections to both component clips before forming a new composite actuator clip.The example considers merging right = (+, 0) and up = (0, +) into a right-up action.
Learning response: After training on horizontal and vertical directions, the agent is confronted with diagonal attacker movements and receives reinforcement for movements into the correct quadrant.The figure description specifies a time-zero transition to diagonal moves and weight updates for right or upward responses.

VI. CONNECTION WITH EXISTING LITERATURE

The paper situates projective simulation among cognitive, machine-learning, and reinforcement-learning approaches that reuse internal experience for prediction and planning. On the simple invasion game, it reports competitive performance with experience replay and Dyna-style planning, while multiple reflections can also raise the maximum blocking efficiency.

Related traditions: Projective simulation is compared with research on cognitive maps, episodic memory, internal simulation, prediction, and reinforcement-learning planning.The comparison spans psychology, cognitive neuroscience, artificial intelligence, machine learning, and robotics.
Experience replay: Unlike experience replay, projective simulation uses explicit episodic representations, emotion-conditioned reflections, structural clip composition, and continuous online interaction with real actions.The paper distinguishes changes to network structure from merely changing transition weights.
Dyna-style planning: Projective simulation resembles Dyna-style planning through fictitious experience, but the paper describes their underlying conceptual frameworks as different.Dyna-style planning generates imaginary experience from a world model before applying model-free reinforcement learning.
Performance comparison: Increasing reflection, replay, or planning parameters increases learning speed across the compared models, with similar performance on the simple invasion game.Figure 18 compares projective simulation, experience replay, and Dyna-style planning under matched initial learning speed and asymptotic value.
Performance comparison: Multiple reflections in projective simulation increase both learning speed and the maximum achievable blocking efficiency, unlike the corresponding experience-replay and Dyna-style conditions.The paper notes that the latter models can achieve the higher maximum by changing the external reward.
Scaling: Learning time scales linearly with the sizes of the percept and action spaces as their dimensions increase, similarly to the compared approaches.The reported scaling concerns |S| and |A|, respectively, in more complex task environments.

VII. QUANTUM PROJECTIVE SIMULATION

The paper generalizes projective simulation from a classical random walk over clips to a quantum walk over clip states in a Hilbert space. Quantum memory supports coherent transitions, superpositions, and interference, while incoherent transitions provide a master-equation extension and can introduce directedness.

Motivation: Quantum projective simulation is motivated by possible quantum-scale agent applications and by the prospect that quantum computation could improve some computational tasks.The paper also warns that quantum or classical computers alone do not make an agent intelligent without continuous environmental feedback.
Quantum representation: In the quantum model, percepts and actions remain classical while clips become quantum states in a memory Hilbert space, and the classical clip walk becomes a quantum walk.An external percept excites a corresponding percept clip as a basis state of the quantum memory.
Quantum transitions: Elementary and composite clip transitions are represented through quantum amplitudes, with composite-transition amplitudes producing quantum interference.The scalar product defines an elementary transition amplitude, while the modulus squared of the composite expression yields interference effects.
Quantum transitions: The clip network is represented as a graph whose vertices are clips and whose edges denote possible transitions, with a Hamiltonian generating the quantum walk.The paper associates a graph G = (V, E) with the clip network before specifying the Hamiltonian.
Dynamics: The quantum master equation generalizes the classical random-walk process: setting H = 0 recovers the classical process, while Hamiltonian and Lindblad terms describe coherent and incoherent transitions.Coherent transitions support superpositions and interference; incoherent transitions can represent spontaneous quantum jumps and directed walks, with a coherence–directedness trade-off.
Agent architecture: A quantum memory call proceeds through memory activation, a quantum walk from the activated state, and measurement-based memory output that induces classical action.The action probability is obtained by measuring actuator observables of the evolving memory state.
Scope and prospects: Quantum superpositions of percept states may enable potentially large speed-ups in exploring memory, although additional quantum computational elements remain outside the paper’s scope.The paper identifies further extensions beyond the simple quantum walk as future work.

VIII. CONCLUSION

The paper presents projective simulation as a learning framework in which agents generate fictitious situations through episodic and compositional memory, influencing future actions. It reports an implemented invasion-game agent and argues that the abstract clip-based random walk can support embodied and quantum extensions.

Core contribution: Projective simulation lets an agent project itself into self-generated fictitious situations that influence future actions.The paper presents these projections as an elementary basis for creative action within learning agents.
Demonstration: The authors implemented and tested a projective-simulation agent in the invasion game using episodic and compositional memory.They describe the examples as simple but concrete demonstrations of learning and simulation.
Future applicability: The authors believe projective simulation could be implemented in autonomous agents or robots operating in realistic task environments.This claim is framed as a potential extension within a comprehensive embodied approach to artificial intelligence.
General framework: The clip network and its random walk are proposed as a physically grounded, implementation-independent framework whose structure grows through clip variation and composition.The paper interprets the walk as propagation of excitations representing information-carrying physical degrees of freedom.