Source-linked AI summary
World models of environment, agent and joint agent-environment systems
Manuel Baltieri, Filippo Torresan, Yivan Zhang, Alexander Boyd, Fernando E. Rosas
TL;DR
The paper addresses the gap created by treating world models mainly as predictors of variables rather than models of distinct interaction channels. It defines canonical environment, agent, and joint models using computational mechanics, then derives support-restricted models from closed-loop coupling. The resulting structure links support-restricted environment states to joint causal states and shows that coupling can make a finite model possible even when the unrestricted environment model is infinite.
Problem
World models are usually distinguished by predicted variables, although environment, agent, and realised joint channels support different predictive questions and causal abstractions.
Method
The paper defines unique, minimal, unifilar canonical models for environment, agent, and joint channels, then constructs support-restricted environment and agent models induced by fixed closed-loop coupling.
Results
The support-restricted environment model is determined by the joint model’s non-sink states and transition structure, while a POMDP/controller example has an infinite unrestricted environment model but a finite induced support-restricted model.
Takeaways & Limitations
The framework clarifies what a world model is a model of and separates predictive states needed for environment planning, policy behaviour, and on-policy interaction.
Takeaways & Limitations
The canonical models are treated in idealised scenarios, with analysis of trained AI models and belief attribution left as future work.
Abstract
from arXiv · showhide
World models are a central component of model-based reinforcement learning. They are usually discussed in terms of what variables they predict, such as observations, rewards, states, latent or information states. We argue that there is a prior distinction: which channel they model. We consider three cases: the environment channel $O_{:} \mid A_{:}$, the agent channel $A_{:} \mid O_{:}$, and the realised joint process $(A, O)_{:}$, equivalently viewed as a channel with no inputs. Using computational mechanics, we define canonical predictive models for these three cases as $ε$-transducers or $ε$-machines. Canonical environment models recover standard predictive state representations, while the other two give analogous notions of canonical models for the agent and the joint system. We then build canonical support-restricted environment and agent models induced by closed-loop coupling, whose predictive equivalences range over continuations supported by the realised interaction. The key structural result is that canonical support-restricted environment states factor through the canonical joint causal states, and their transition structure is induced directly from the joint model; the agent-side construction is dual. Finally, we give a POMDP/controller example in which the unrestricted environment model has infinitely many states while the canonical support-restricted model induced by the coupling is finite. The framework clarifies what different world models are models of, and how coupling and support restriction can change their canonical predictive structure and complexity.
1 Introduction
The paper argues that world models should be distinguished by the channel they model, not only by the variables they predict. It develops canonical models for environment, agent, and realised joint channels, then characterises support-restricted models induced by closed-loop coupling.
- Channel-based distinction: World models can represent the environment channel, agent channel, or realised joint action-observation process, each answering a different predictive question.These channels correspond respectively to future observations under actions, future actions under observations, and traces generated by the coupled interaction.
- Channel-based distinction: Because the channels induce different causal equivalence relations, they also yield different abstractions, model complexities, and use cases.A state sufficient for predicting observations under actions need not predict the agent’s actions or the realised interaction.
- Canonical models: The framework uses computational mechanics to define canonical predictive models as ε-machines or ε-transducers for all three channels.The environment construction recovers predictive state representations, while analogous canonical models describe policy behaviour and coupled interaction.
- Support restriction: Closed-loop coupling induces support-restricted environment and agent models whose predictive equivalences range only over continuations supported by the realised interaction.This distinguishes policy-conditioned prediction from counterfactual queries outside the interaction’s support.
- Support restriction: The joint model determines the non-sink states and transition structure of the support-restricted environment model, showing that compact closed-loop prediction need not imply a compact unrestricted model.Unsupported queries are represented by an added sink state.
2 Canonical models
The paper defines canonical world models by the channel they model: environment, agent, or realised joint process. These models use causal-state predictive equivalences, while support restriction captures only continuations realised by a closed-loop coupling and can change model complexity.
- Predictive channels: Three channels define distinct predictive questions: observations given actions, actions given observations, and realised action-observation traces.The joint process is treated as a null-input channel.
- Canonical models: Canonical environment, agent, and joint models are defined with computational-mechanics causal states as minimal predictive abstractions for their respective channels.The models are represented as ε-transducers or ε-machines and are unique, unifilar, minimal, and predictive in the computational-mechanics sense.
- Canonical models: The canonical environment model recovers the all-tests, presentation-independent semantics of predictive state representations.Environment tests supply future actions and predict corresponding future observations.
- Canonical models: The canonical agent model represents histories by how the agent would act under all possible future observation continuations.Its predictive tests fix future observations and predict future actions.
- Support restriction: Support-restricted models evaluate only observation or action continuations supported by the realised coupling, distinguishing policy-conditioned prediction from counterfactual environment modelling.The support-restricted environment and agent models induced by the coupling factor through the joint model.
- Relations among models: In the running example, the unrestricted environment model is infinite, whereas the agent and joint models are finite; separately defined causal states need not preserve joint correlation, timing, or support information.The joint model therefore cannot generally be obtained by simply identifying or combining the environment and agent state spaces.
3 Canonical support-restricted models from closed-loop coupling
Support-restricted models limit predictive equivalence to continuations supported by a realised closed-loop interaction, totalising unsupported queries with a sink. Their environment-side states and transitions are induced by the canonical joint model, allowing finite support-restricted structure even when unrestricted environment structure is infinite.
- Support restriction: Support-restricted models compare predictions only over future continuations supported by the realised coupling, unlike unrestricted models that include arbitrary counterfactual continuations.The same restriction is applied dually to the agent model by exchanging actions and observations.
- Support restriction: For deterministic policies, the one-step realisable action set contains exactly the on-policy action.More generally, an action is supported when the policy assigns it positive probability and some observation has positive conditional probability.
- Totalisation: Unsupported action queries emit ⊥, after which the totalised environment channel remains in a single sink state.On supported actions, the channel agrees with the ordinary conditional observation law induced by the joint process.
- Relation to the joint model: Every non-sink support-restricted environment state is obtained from a canonical joint causal state through a unique surjective map.Thus ordinary support-restricted states are quotients of joint causal states, with at most one additional totalisation sink.
- Relation to the joint model: The canonical joint model determines the support-restricted environment model’s state space and transition family.The interface fixes the action alphabet, totalisation adds ⊥, and the joint model determines non-sink states and transitions.
- Example: In the example, the support-restricted environment model has six states including ⊥, despite an unrestricted environment model with infinitely many states.The controller bounds supported hold continuations, producing finite closed-loop predictive structure.
4 Related work
Related work has largely focused on environment-channel world models, while adjacent literatures connect predictive models to several abstractions and agent-side perspectives. The paper separates these environment models from agent and joint models and relates support restriction to on-policy abstractions.
- Environment-channel models: Computational-mechanics treatments in machine learning have mostly represented environment-channel predictive structure with machines or transducers.These connect to observable operator models, predictive state representations, belief MDPs, and bisimulation-style abstractions.
- Agent-side and on-policy perspectives: Agent-side predictive models relate to self-predictive AI, policy distillation, plasticity, habit-formation models, and history-dependent policies.The related-work discussion also connects support restriction to adaptive offline reinforcement learning and on-policy bisimulation.
5 Conclusions
The paper characterizes world models by the channel they model and uses computational mechanics to define canonical environment, agent, and joint models. It also connects these models to predictive-state, action-predictive, policy-induced, and belief-based abstractions.
- Canonical world models are unique, minimal, and unifilar models organized by whether they represent the environment, agent, or realized joint process.
- Environment models connect to predictive state representations and belief-MDP abstractions, while agent and joint models connect to action prediction, policy distillation, controllers, and embedded agency.
- Computational mechanics defines causal states as history classes with identical predictive consequences, yielding minimal sufficient statistics for future prediction.
- For input-output channels, ε-transducers provide canonical minimal unifilar presentations whose states predict future outputs from histories and inputs.
- The framework assumes stationary causal channels and distinguishes fixed-input output processes from joint processes induced by an input process.
B Proofs
The proofs establish that support-restricted environment states are determined by joint causal states and that their transitions are inherited from the joint model. Totalisation adds an absorbing sink for unsupported queries.
- Joint causal equivalence implies equal finite-dimensional laws for the totalised support-restricted environment channel.
- A map from joint causal states to non-sink support-restricted environment states is well defined because equivalent joint histories induce the same support-restricted causal state.
- Unsupported queried actions emit the totalisation symbol and move to an absorbing sink, while supported actions use normalized joint one-step probabilities.
- The support-restricted transition branch, output probabilities, and successor state do not depend on which joint-state representative is chosen.
- The complete support-restricted environment transducer consists of the joint-state quotient as non-sink states plus the totalisation sink and induced transitions.
C Running example
The running example couples a binary hidden-state environment to a deterministic finite-state controller with three memory modes. It separates unrestricted, joint, and support-restricted model structures.
- The example uses binary actions and observations, a hidden binary environment state, and a controller memory state with modes α, β, and γ.
- The example distinguishes the unrestricted environment, agent, realized joint, and canonical support-restricted environment models.
- The environment resets its hidden state to a fair coin under action 0 and holds it fixed under action 1.
- Observations are noisy readouts of the updated hidden state, with error parameter η.
- The controller resets in mode α, holds in mode β, and conditionally enters mode γ before returning to reset.
C.2.1 Canonical environment model
The canonical unrestricted environment model tracks filtering beliefs generated by action-observation histories. Repeated holds produce infinitely many distinguishable beliefs and therefore infinitely many causal states.
- The model represents the posterior belief pt = Pr(Et = 1 | H:t = h:t) after an action-observation history.
- For queried action a, the predictive prior qt(a) is used to update the posterior according to the observed outcome.
- Under the hold action, the probability of observing 1 is an affine, strictly increasing function of the current belief.
- Distinct beliefs are separated by the counterfactual one-step input At = 1 and therefore correspond to distinct environment causal states.
- Infinitely many reachable posterior beliefs imply that the unrestricted canonical environment model has infinitely many causal states.
C.2.2 Canonical agent model
The canonical agent model treats observations as inputs and distinguishes controller memory modes by the future action streams they induce. In the example, three modes are pairwise predictively distinct.
- The deterministic controller has three causal states corresponding to memory modes α, β, and γ.
- Mode α differs from β and γ through its current action, emitting 0 instead of 1.
- Modes β and γ differ under observation 1 because they transition to different modes and emit different next actions.
C.2.3 Canonical joint model
The canonical joint model represents the realised action-observation process using predictive configurations that combine controller mode and reachable belief values. Its five states are unifilar and pairwise predictively distinct.
- The joint model predicts the realised coupled process rather than observations or actions under external inputs.
- Predictive configurations are determined by controller mode together with belief values occurring under the realised coupling.
- The realised joint process has five predictive states with transition probabilities parameterised by c, d, and g.
- The presentation is unifilar: the current state and emitted symbol determine the next state.
- The five states are pairwise predictively distinct because their emissions, observation probabilities, or successor structures differ.
C.3.1 Canonical support-restricted environment model
Support restriction evaluates environment predictions only on action continuations supported by the realised controller. This can reduce an infinite unrestricted model to a finite canonical model with a totalisation sink.
- The unrestricted environment model is infinite because arbitrary future action continuations include arbitrarily long hold sequences.
- Under the realised controller, hold continuations are bounded because the controller returns to α after at most two consecutive holds.
- Each ordinary support-restricted environment state is the image of a joint causal state, with the five non-sink images pairwise distinct.
- Unsupported queried actions transition to the totalisation sink sE⊥, while ordinary transitions are induced from the joint presentation.
- The canonical support-restricted environment model has exactly six causal states.
- In this example, the support-restricted agent model retains three ordinary causal states and adds a sink because every finite observation continuation has positive probability.
- This agent-side outcome is specific to the example; environmental support restrictions can generally merge or alter support-restricted agent states.
D Relation between the three canonical models, with extra structure
For compatible factorised environment and agent models, reachable product states provide a sufficient presentation of the realised joint process. The canonical joint model is then a quotient of that reachable presentation, subject to an explicit joint-sufficiency assumption.
- The construction assumes compatible environment and agent models whose closed-loop coupling defines a stationary joint process.
- The product state is required to be sufficient for predicting the realised joint future, although it need not itself be a joint causal state.
- Reach(SE × SM) contains the product states occurring with positive probability and carries transition dynamics for the joint presentation.
- The canonical joint model is a quotient of the reachable product presentation, yielding a surjective map χ onto its causal states.
- Any finite product-state bound applies only to an explicitly specified factorised presentation satisfying joint sufficiency.