On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

1 Introduction to Reinforcement Learning (RL) with Recurrent Neural Networks (RNNs) in Partially Observable Environments1

• Partially observable RL requires agents to maximize cumulative reward despite unknown dynamics, delayed consequences, and incomplete observations.Source: §1 · p. 2
• RNNs can represent computable interactions with dynamic environments by storing input histories, but their weights must be learned through trial and error.Source: §1 · p. 3

1.1 RL through Direct and Indirect Search in RNN Program Space

• Direct evolutionary or policy-gradient search can train small RNNs, but network-sized search spaces remain too large for complex high-dimensional control from scratch.Source: §1.1 · p. 3
• Compressed encodings enabled evolution of RNN controllers with over a million weights from high-dimensional video, including control and visual processing learned from scratch.Source: §1.1 · p. 3

1.2 Deep Learning in NNs: Supervised & Unsupervised Learning (SL & UL)

• Backpropagation can suffer exponentially vanishing or exploding cumulative errors, motivating architectures and unsupervised methods that improve learning across long temporal dependencies.Source: §1.2 · pp. 3–4

1.3.1 Early RNN Controllers with Predictive RNN World Models

• Predictive world models let RL controllers rehearse actions through cheap simulated trials, but earlier systems mainly simulated futures step by step and lacked abstract hierarchical reasoning.Source: §1.3.1 · p. 5

1.4 Hierarchical & Multitask RL and Algorithmic Transfer Learning

• Hierarchical RL decomposes tasks, yet existing approaches use limited fixed credit-assignment schemes rather than problem-specific mechanisms that adapt themselves.Source: §1.4 · p. 6

2 Algorithmic Information Theory (AIT) for RNN-based AIs

• RNNAIs are designed to learn predictive environmental models and use them for hierarchical planning, abstract reasoning, and increasingly general problem solving.The paper characterizes this capability as “learning to think.”Source: §2 · p. 6

2.1 Basic AIT Argument

• AIT defines the mutual information between programs through K(q | p), the shortest program that computes q given p, and predicts faster search when this conditional program is short and efficient.The argument assumes q solves a target problem and that the shared information can be exploited instead of rediscovered from scratch.Source: §2.1 · p. 7

2.2 One RNN-Like System Actively Learns to Exploit Algorithmic Information of Another

• A controller C can learn to query and reuse the algorithmic information in a pretrained RNN-like model M while optimizing its own parameters wC.M’s parameters wM are frozen, whereas C learns through trials and an interface that transmits queries and answers.Source: §2.2 · p. 7
• When M contains substantial task-relevant information, the interface extracts it quickly, and wC is small relative to fixed wM, C’s search space should be much smaller than a competing controller’s search from scratch.This is an AIT-based prediction rather than an experimental result in this report.Source: §2.2 · pp. 7–8

2.3 Consequences of the AIT Argument for Model-Building Controllers

• The proposed model-building controllers use a larger unsupervised RNN M to model interaction histories, while C accesses M for planning and may support hierarchical reuse, analogy, and other computable forms of abstract reasoning.The paper also presents C-generated experiments as a route to efficiently improving M.Source: §2.3 · p. 8

3 The RNNAI and its Holy Data

• At each time step, sensory and reward inputs are combined, actions affect future observations, and the objective is to maximize cumulative reward.The history also records the concatenated sensory, reward, and action vectors.Source: §3 · p. 8
• The RNNAI preserves its entire lifelong history of observations, actions, and rewards, including data from failed trials and apparent noise.This “holy data” supports later retraining and possible discovery of regularities.Source: §3 · pp. 8–9

3.1 Standard Activation Spreading in Typical RNNs

• RNN-like implementations may include LSTMs, fast-weight networks, meta-learners, or CNN-LSTM combinations for processing large inputs such as video.The paper uses “RNN” broadly for these general-computing architectures.Source: §3.1 · p. 9
• RNN programs are determined by recurrently reused weights that propagate activations through causal events and can encode memory across time.Weight sharing across space or time can reduce descriptive complexity.Source: §3.1 · p. 9

3.2 Alternating Training Phases for Controller C and World Model M

• The system alternates controller learning on recorded trials with world-model retraining on the growing interaction history to avoid training instabilities.Algorithm 1 freezes M while C learns, then unfreezes M for a sleep-phase update before repeating.Source: §3.2 · pp. 9–10

4 The Gradient-Based World Model M

• M receives sensory, reward, and action histories and predicts the next sensory vector, providing a predictive model of the agent’s interactions.Its outputs are trained from the recorded history rather than only from successful trials.Source: §4 · p. 10, §4.1 · p. 10

4.1 M’s Compression Performance on the History so far

• M’s compression score is the bits needed to encode the model plus the bits needed to encode deviations between predictions and observations.This penalizes models that fit noise through excessive complexity rather than minimizing prediction error alone.Source: §4.1 · pp. 10–11
• Prediction errors are assigned code lengths through an error distribution, while model complexity can be estimated from nonzero weights or weight-value code lengths.These practical estimates are easy to calculate but do not fully capture short programs describing the entire weight matrix.Source: §4.1 · p. 11

4.2 M’s Training

• As the history grows, M is expanded or pruned and retained only when retraining improves bitsM + bitsH.Sequential network construction changes the model’s size during training.Source: §4.2 · p. 11
• M can be retrained by replaying the full history or selected trials, with activation storage required when trial states are not reset.The selected-trial option reduces the amount of history replayed during a sleep phase.Source: §4.2 · p. 11

4.3 M may have a Built-In FNN Preprocessor

• A CNN or other feedforward autoencoder can compress each sensory frame before an RNN predicts the compressed code, with reconstruction included in total compression evaluation.This preprocessing reduces redundancy in large inputs such as video.Source: §4.3 · p. 11

5.1 C as a Standard RL Machine whose States are M’s Activations

• C can use M’s hidden activations and predictions as a Markov-style state for standard RL, separating history prediction from reward optimization.The controller receives the constructed state, chooses an action, and M updates its hidden state and prediction before the action affects the environment.Source: §5.1 · p. 12

5.2 C as an Evolutionary RL (R)NN whose Inputs are M’s Activations

• The same M-derived state can feed an evolutionary or policy-gradient controller whose weights are trained while M remains a predictive preprocessing system.This variant uses the controller’s learned mapping from M’s activations to actions.Source: §5.2 · p. 12

5.3 C Learns to Think with M: High-Level Plans and Abstractions

• Earlier model-based systems were limited because noisy environments require approximate predictions and conventional planning simulates alternative futures step by step.The proposed controller instead learns problem-specific ways to access useful parts of M.Source: §5.3 · pp. 12–13
• The main proposed CM system lets a small controller query, edit, or invoke parts of a large fixed world model through learned connections.Only C’s comparatively few connections change during RL training, while M’s many connections remain fixed.Source: §5.3 · p. 13
• The approach is expected to reduce controller search when M shares time-bounded mutual algorithmic information with the task solution, though exploiting M itself still costs search.The paper contrasts this with earlier transfer learning where reused recursive code sped search by a factor of 1000.Source: §5.3 · pp. 13–14
• C can learn to exploit abstract units and hierarchical representations in M, potentially shortcutting long input sequences to anticipate useful or harmful outcomes.The approach is intended to support abstract hierarchical planning and analogy-based reasoning rather than only step-by-step future simulation.Source: §5.3 · p. 13
• C may learn when to attend to M, ignore environmental inputs, or use M for internal computation through a special gating unit.This allows M to support “thinking” without always processing ordinary environmental input.Source: §5.3 · p. 14

5.4 Incremental / Hierarchical / Multitask Learning of C with M

• Incremental and hierarchical variants freeze previously learned controller components and add new units or connections for later tasks, reducing the new task’s search space.The paper also describes coefficient-based incremental search and growing runtime limits for tested programs.Source: §5.4 · p. 14

• M can discover compact predictive representations, such as gravity or three-dimensional spatial structure, that reduce the information needed to encode observations.Source: §6 · pp. 14–15
• Predictive coding explains mirror neurons as shared feature detectors encoding mutual algorithmic information across similar observed and performed actions.Source: §6 · p. 15
• Intrinsic reward encourages C to generate experiments whose resulting data improve M’s compression performance.Source: §6 · p. 15
• Evaluating compression over the entire history is impractical, so the heuristic emphasizes recent-trial improvements and periodically retrains on earlier trials to avoid catastrophic forgetting.Source: §6 · p. 15
• A self-modularizing M enables localized retesting and credit assignment, while verification-time penalties favor experiments producing quickly checkable compression progress.Source: §6 · p. 15

• RNNAIs combine an RL controller C with an RNN-based predictive world model M, enabling C to selectively probe and reuse M’s internal programs.Source: §7 · p. 15
• The architecture supports continual acquisition of problem-solving procedures, including self-invented tasks that resemble a scientist’s search for novel, compressible data.Source: §7 · p. 15

• RNN controller C steers the agent through trials, while world model M predicts new inputs from histories of previous inputs and actions.Source: Figure 1 · p. 36
• C may query and exploit M’s program-encoded regularities to search for rewarding behavior more cheaply than learning reward-generating programs from scratch.Source: Figure 1 · p. 36
• C can also receive intrinsic reward for experiments that produce data with unknown regularities and improve M’s compression.Source: Figure 1 · p. 36