Source-linked AI summary
The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning
Marko Cvjetko, Benedikt Hartl, Michael Levin, Clément Moulin-Frier, Pierre-Yves Oudeyer
TL;DR
Existing cellular-automata exploration is largely open loop, limiting interaction with complex evolving systems. The paper introduces autotelic, goal-conditioned RL and instantiates it as CARL on Lenia, where agents discover and control solitons through local interventions. CARL generalizes across conditions and supports real-time human guidance through high-level commands.
Problem
Existing cellular-automata exploration mostly fixes initial conditions and runs simulations without intervention, while controlling partially understood complex systems remains challenging.
Method
CARL trains a goal-conditioned autotelic RL policy to observe Lenia and apply minimal, local perturbations toward diverse goals, including intervention-cost constraints.
Results
CARL discovers solitons across update rules, steers their movement, generalizes zero-shot to out-of-distribution conditions, and enables real-time human maze guidance.
Takeaways & Limitations
Trained policies can be reused as real-time interfaces that combine autonomous soliton discovery, adaptive control, and human high-level commands.
Takeaways & Limitations
The framework depends on domain expertise for reward, action-space, and observation design, and Lenia’s favorable observability and determinism may not hold in biological systems.
Abstract
from arXiv · showhide
Existing methods for exploring cellular automata and other complex systems mostly operate in open loop: they set initial conditions, execute a full simulation, and observe the outcome, without intervening during execution. We introduce a closed-loop framework based on autotelic reinforcement learning, in which an agent autonomously samples diverse goals and learns a goal-conditioned policy to intervene in a complex system through minimal, local perturbations. We instantiate this framework on Lenia, a continuous cellular automaton known for life-like self-organizing patterns, in an agentic system we call CARL, and demonstrate three capabilities. First, CARL discovers stable solitons across a wide range of Lenia update rules at a higher rate than heuristic baselines. Second, it learns to steer the movement direction of existing solitons with few interventions, showing that CARL can control self-organizing patterns, not only create them. Third, humans can use trained agents to guide solitons through maze environments in real time by specifying high-level directional commands that the agent translates into low-level interventions. Trained across diverse goals, update rules, and random initial states, the agents acquire policies that generalize zero-shot to various out-of-distribution conditions. These results suggest a path toward artificial experimentalist agents that, autonomously or with human guidance, discover and control emergent phenomena in complex systems.
Introduction
The paper addresses the challenge of controlling partially understood complex systems by proposing closed-loop, autotelic reinforcement learning. CARL observes cellular automata during evolution, learns diverse goals, and applies interventions to discover and control self-organizing phenomena.
- Complex-system research seeks both to explain emergent phenomena and to control systems toward desired states with minimal effort.
- Controlling complex systems is difficult because behavior arises from multiscale interactions, nonlinearities, feedback loops, and delayed effects.
- Most existing cellular-automata exploration operates open loop, fixing parameters and initial conditions without interaction during rollout.
- The proposed autotelic RL framework autonomously learns to achieve diverse goals and translates desired experimental outcomes into interventions.
- CARL uses closed-loop observation and intervention to shape cellular-automata dynamics over time.
- Experiments on Lenia show discovery and control of self-organizing phenomena with limited interventions, sample efficiency, out-of-distribution generalization, and real-time human guidance.
- Although demonstrated on Lenia, the framework is designed to be system-agnostic and extendable to more biologically grounded models.
Related Work
Prior work has explored cellular-automaton behavior through search, optimization, diversity-driven discovery, and some forms of intervention. The paper situates CARL among approaches that move beyond passive open-loop exploration toward reactive control.
- Autotelic approaches have also been applied to automated protocell-behavior discovery and gene-regulatory-network exploration.
- Cellular automata model processes across ecology, urban development, and physics while generating complex phenomena from simple local updates.
- Existing exploration methods include random search, manual tuning, hand-crafted heuristics, gradient optimization, novelty search, quality-diversity, and intrinsically motivated goal exploration.
- Some studies apply external forces, optimize rules at planned checkpoints, or train agents for specific cellular-automaton management tasks.
Method
The framework trains a goal-conditioned policy to interact with a simulatable complex system through small interventions while pursuing sampled trajectory, system, and effort goals. CARL instantiates this loop on Lenia using localized perturbations, soliton detection, and a spatially structured RL policy.
- General Framework: The framework has an autotelic discovery phase followed by real-time use of the learned goal-conditioned policy to produce and control self-organizing patterns.
- General Framework: A complex system is represented by a state space and update rule, with no requirement beyond the ability to simulate its dynamics.
- General Framework: Interventions modify the current state through small perturbations, optionally including a no-op action rather than rewriting the whole system.
- General Framework: Goals can specify target behavior, update rules, trajectory properties, and intervention effort through a sampled goal space and reward function.
- General Framework: Each episode samples a goal and initial state, observes recent system states, selects an action, applies an intervention, evolves the system, and receives a reward.
- General Framework: During inference, dynamically changing goals allow humans to issue high-level commands that the policy translates into low-level interventions.
- Instantiation: Lenia: Lenia evolves a periodic grid through convolution with a kernel, an element-wise growth function, and a step size.
- Instantiation: Lenia: Solitons are localized Lenia patterns that persist and often move; detection follows 5000 uninhibited update steps and a mass-based classification filter.
Experiments
CARL is evaluated as a closed-loop agent for mass tracking, soliton creation, generalization across modified conditions, directional steering, and human-guided control in Lenia. Across these experiments, it creates solitons more effectively than baselines, generalizes to novel settings, and supports real-time redirection, while failures occur under some collisions and high action costs.
- Experimental setup: CARL samples target masses, action costs, and update rules to learn one policy that balances mass tracking against intervention effort.The reward penalizes target-mass deviation and non-trivial interventions, while action costs encourage self-sustaining configurations.
- Mass tracking evaluation: The agent tracks target mass across most evaluated conditions, but performance degrades at extreme targets and high action costs.At low targets mass dissipates, while high targets can grow unboundedly, requiring costly intervention.
- Soliton creation: Solitons form at high rates especially for target masses τ = 100 and 150, with generated mass correlating with the target.For high targets, CARL can create multiple independent solitons whose combined mass matches the target.
- Soliton creation: CARL outperforms all baselines in overall soliton creation rate and in the number of update rules producing at least one soliton.Mass-based heuristics share mass information with CARL but lack spatial awareness, while the no-op baseline rarely produces solitons.
- Generalization: The trained agent generalizes to modified action parameters, kernel radii, and unseen update-rule spaces, although success varies across kernels and drops more sharply for down-scaled kernels.It identifies regions of novel (µ, σ) spaces supporting soliton formation and remains above the no-op baseline across tested radii.
- Soliton direction and human-in-the-loop control: Direction steering achieves mean cosine similarity of 0.91 ± 0.14 on training conditions and 0.76 ± 0.15 when both solitons and directions are held out.In maze control, dynamically changing goals redirect solitons repeatedly; lower action costs increase intervention frequency and speed, creating a speed-precision trade-off.
Discussion
CARL combines autonomous discovery with sparse, closed-loop control of Lenia solitons, generalizing across out-of-distribution conditions and supporting real-time human guidance. The framework points toward artificial experimentalists that can compose autonomous and human-directed interventions, while remaining bounded by domain expertise and Lenia’s favorable assumptions.
- Framework: CARL uses a goal-conditioned policy to observe evolving systems and apply minimal, local perturbations toward diverse goals.The framework replaces passive open-loop observation with closed-loop intervention.
- Capabilities: CARL discovers solitons across update rules and initial states, controls their movement direction, and supports real-time maze guidance through simple human commands.The system serves both autonomous discovery and interactive control.
- Sparse intervention: Action costs encourage sparse control: agents briefly redirect solitons and then let their intrinsic dynamics sustain the new heading.This design also makes self-sustaining soliton formation an economical solution in the mass-tracking task.
- Generalization and reuse: CARL generalizes across mass targets, action parameters, system dynamics, and novel update rules, enabling policy reuse and composition for maze navigation.The maze task combines soliton creation with adaptive redirection and allows a human to modify goals in real time.
- Future direction: CARL is presented as a first step toward artificial experimentalists that autonomously decide what to investigate and combine actions through combinatorial repurposing.The proposed direction includes symbiotic structuring of actions and hierarchical control.
- Limitations: The framework’s instantiation requires domain expertise, and Lenia’s full observability, determinism, and simple action space may not hold in biological or biomedical settings.Extending to partial observability, stochasticity, and high-dimensional action spaces is identified as a central future challenge.