AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretability

• Synthetic world models enable controlled evaluation of AI agents before deployment, including tests of edge cases and novel situations without real-world consequences.Source: §1 · p. 2
• High-fidelity simulations can impose heavy computational costs that restrict the breadth and depth of safety and reliability assessments.Source: §1 · p. 3
• The paper bridges reinforcement learning, control theory, and computational mechanics to identify a fundamental trade-off between world-model efficiency and interpretability.Source: §1 · p. 3
• It distinguishes forward interpretability for predictive capabilities from reverse interpretability for tracing undesirable outcomes.Source: §1 · p. 3

• World models for sandboxing should be chosen according to the evaluation purpose rather than treated as a single universally optimal representation.Source: §2 · p. 4
• The ‘AI in a vat’ perspective considers all world models indistinguishable from the agent’s perspective before selecting among them.Source: §2 · p. 4
• Computational efficiency targets minimal resources for evaluating agent behaviour or providing formal capability guarantees.Source: §2 · p. 4
• Forward interpretability studies which features agents can learn and how representations are encoded inside them.Source: §2 · p. 4
• Reverse interpretability runs models backward to investigate origins and tipping points leading to desirable or undesirable outcomes.Source: §2 · p. 4

3.1 World models

• An interface specifies outcome distributions for action sequences and is anticipation-free when outcomes up to time t depend only on actions through time t.Source: §3.1 · p. 4
• World models encode information linking past events to present outcomes while enforcing that future actions cannot affect previous states or outcomes.Source: §3.1 · p. 5

3.2 Transducers

• A transducer is a memory-state process whose Markov kernel jointly generates outputs and next states from the current input and memory state.Source: §3.2 · p. 5
• Transducers generate anticipation-free interfaces when their state dynamics satisfy a Markov condition conditioned on current state and history.Source: §3.2 · p. 5
• Memoryless and fully observable interfaces are special cases, while non-Markovian dynamics require more elaborate world models.Source: §3.2 · p. 6
• Input-Moore, output-Moore, and I-O Moore transducers impose progressively structured restrictions on whether outputs or state updates use current inputs and outputs.Source: §3.3 · p. 6

4 Minimal world models

• The full-history world model generates the interface but is not parsimonious, motivating reductions defined as state mappings that preserve outcome statistics and Markovian dynamics.All interfaces have at least one transducer presentation, allowing minimisation to be studied within the transducer framework.Source: §4.1 · pp. 6–7

4.2 Reduction via bisimulation

• Bisimulation-based coarse-graining merges equivalent states while preserving the interface, yielding a reduced transducer for the same action-outcome process.World-model reductions correspond exactly to bisimulations, so coarse-graining all bisimilar states achieves a full reduction of the given transducer.Source: §4.2 · p. 7
• Bisimulation reductions need not reach a global minimum because they cannot generally remove linear dependencies among states without distorting the interface.Relaxing state-merging criteria through bisimulation metrics introduces interface distortion rather than solving this limitation.Source: §4.2 · pp. 7–8
• A transducer with canonical dimension d(T) = n cannot be replaced by any interface-equivalent transducer with fewer than n memory states.Canonical dimension therefore identifies when a given transducer representation is already irreducible in state count.Source: §4.2 · p. 8

4.3 Reduction via pseudo-probabilities and generalised transducers

• Generalised transducers replace non-negative probabilities with quasi-probabilities, enabling linear reductions to fewer states whenever their canonical dimension is below the original state count.These models can always be reduced and admit practical reduction algorithms, but the resulting world model is opaque and sacrifices interpretability.Source: §4.3 · pp. 8–9

5.1 Beliefs as world models

• Belief transducers represent world-model states as distributions over underlying states, extending transducers into belief-state spaces while preserving the original interface when faithful.A faithful belief transducer generates the same interface as its underlying transducer.Source: §5.1 · pp. 9–10
• Predictive Bayesian beliefs form unifilar belief transducers and generate the same interface as the original transducer when initialized with the original state distribution.Their belief state is bt(st) := Pr(St = st|H:t−1 = h:t−1).Source: §5.1 · pp. 9–10
• Predictive and postdictive Bayesian beliefs provide related constructions for different temporal inference directions, with postdictive beliefs using current outputs to infer the present state.For I-O Moore transducers, postdictive beliefs are faithful when initialized with d0 = p(s0).Source: §5.1 · p. 10

5.2 Minimal predictive world models

• The ϵ-transducer is the minimal predictive model that generates a given interface, obtained by bisimulating any predictive world model.It is unique up to isomorphism and provides an observable representation over potentially fewer states.Source: §5.2 · p. 10
• The ϵ-transducer distils information relevant to predicting future events, supporting evaluation of what agents can learn through an interface.Postdictive models reduce analogously to a time-shifted ϵ-transducer, but that development is deferred to future work.Source: §5.2 · p. 11

6.1 Reversible transducers

• Reversible transducers can be run backward to analyse prior events leading to an undesirable state and to study how outputs lead to actions and identify tipping points.This reverse operation updates world states from sτ+1 to sτ while generating the same interface.Source: §6.1 · p. 11
• Not all transducers are reversible because reversing time can violate the anticipation-free condition required for a valid world-model transducer.Retrodictive constructions therefore extend reverse analysis beyond interfaces that admit exact reversal.Source: §6.1 · p. 11, §6.2 · p. 12
• A transducer is reversible exactly when p(sτ|sτ+1, a:T) = p(sτ|sτ+1, aτ), allowing a reverse kernel that preserves the same interface.The condition must hold for every τ in the considered time range.Source: §6.1 · p. 11
• Figure 4 illustrates reversible transducers as state-transition graphs whose arrows encode actions, outputs, and transition probabilities.For example, 1|0:0.5 denotes Pr(St+1 = s1, Yt = 1|At = 0, St = s0) = 0.5.Source: §6.1 · p. 11, Figure 4 · p. 12

6.2 Retrodictive beliefs

• The BDMSM provides a matrix-based route for calculating predictive and retrodictive Bayesian beliefs and their dynamics from action-outcome sequences.Its forward and reverse views yield bτ and rτ through matrix multiplication with the all-ones vector.Source: §6.2 · p. 12

6 Reverse interpretability via retrodictive world models

• Retrodictive beliefs infer likely past world states from future actions and outcomes, helping identify behavioural origins and potentially favourable or dangerous initial conditions.They provide an analogue of the backward pass of Bayesian smoothing.Source: §6.2 · p. 12

• The framework supports agent-agnostic evaluation without assumptions about an agent’s policy, architecture, or capabilities, informing core world-model design choices.It unifies perspectives from reinforcement learning, control theory, and computational mechanics.Source: §7 · p. 13
• The framework reveals a fundamental trade-off: generalised transducers minimise implementation efficiency but produce opaque, unsampleable quasi-probabilistic world models.The efficiency gain therefore comes at the cost of interpretability.Source: §7 · p. 13
• The ϵ-transducer is the unique minimal real-time world model, capturing all predictive information available through an environment interface.Its uniqueness implies that optimal predictive agents’ belief refinement eventually reaches this model, regardless of their starting world model.Source: §7 · p. 13
• Retrodictive world models support retrospective analysis of undesirable events, including identifying danger zones likely to precede undesirable outcomes.This complements forward interpretability, which evaluates prediction and planning of future events.Source: §7 · p. 13
• The analysis assumes perfect reconstruction, motivating future work on approximate homomorphisms, bisimulation, rate-distortion trade-offs, and compositional world structure.These approaches are identified as possible routes toward more efficient modelling beyond the current scope.Source: §7 · p. 13

A Example of an interface and multiple world models

• The card-deck interface can be represented using roughly 132 epistemic states instead of 13! physical deck states, while retaining all information relevant to the colour-limited agent.The reduced model tracks red and black cards observed since the last shuffle.Source: A Example of an interface and multiple… · p. 20

Supplementary Materials

• Bayesian sufficiency means that the original variable and the target are conditionally independent given the statistic, so the statistic preserves the relevant information.This is expressed as Pr(X = x|Y = y, T = t) = Pr(X = x|T = t).Source: B Sufficient statistics · p. 21
• Minimal sufficient statistics compress information without losing relevance for the target, and their construction parallels bisimulation and the ϵ-transducer equivalence relation.The appendix explicitly links these constructions while distinguishing minimality from arbitrary sufficiency.Source: B Sufficient statistics · p. 21
• A process is a valid transducer memory state exactly when it satisfies either the stated conditional-factorisation condition or the equivalent conditional-mutual-information conditions.The proof establishes both directions of this characterisation.Source: D Alternative characterisations of a… · pp. 22–23, Part 2: Equivalence between conditions… · p. 24

E Proof of Lemma 3

• An interface is fully observable exactly when p(y_t+1|y_:t, a_:) = p(y_t+1|y_t, a_t), while memorylessness requires factorisation into per-time-step output distributions.The proof uses the candidate state S_t = Y_t for full observability and S_t = 0 for memorylessness.Source: E Proof of Lemma 3 · p. 24
• A POMDP induces a transducer because its joint dynamics satisfy the transducer factorisation condition.The POMDP tuple specifies world states, actions, observations, transitions, emissions, and rewards.Source: F Relationship between transducers and… · p. 25
• The standard POMDP presentation corresponds specifically to an I-O Moore transducer.Figure 5 contrasts this form with Mealy, output-Moore, and input-Moore transducers.Source: F Relationship between transducers and… · p. 25, Figure 5 · p. 25

G Proof of Lemma 4

• Using S_t = H_t−1 and S_0 = 0, the proof establishes the factorisation required for a transducer presentation.The argument decomposes the next-state/output distribution into terms derived from the history-conditioned interface probabilities.Source: G Proof of Lemma 4 · p. 25
• The resulting dynamics satisfy p(s_t+1, y_t|s_:t, h_:t−1, a_t:) = p(s_t+1|s_t, h_t)p(y_t|s_t, a_t) = p(s_t+1, y_t|s_t, a_t).This is the condition used to show that the history state is a valid transducer memory.Source: G Proof of Lemma 4 · p. 26

H Proof of Lemma 5

• The resulting reduced transducer remains valid and generates the same interface as the original world model.Source: H Proof of Lemma 5 · p. 26, I Proof of Prop. 1 · p. 27
• A homomorphism-induced state reduction defines a bisimulation, and every bisimulation induces a reduction through equivalence classes.The two directions establish equivalence between transducer reductions and bisimulations, with the reduced model preserving the same interface.Source: I Proof of Prop. 1 · pp. 26–27
• For finite-state transducers, an SVD identifies the nonzero singular-vector subspace used to define new world states and quasi-stochastic matrices.Source: J Algorithms to reduce a transducer · p. 27
• The SVD-based representation is minimal.Source: J Algorithms to reduce a transducer · p. 27

K Proof of Prop. 2

• Predictive Bayesian beliefs form a belief transducer whose generated interface matches that of the original transducer.The equality follows by recursively applying the belief update and initializing the belief with the original state distribution.Source: K Proof of Prop. 2 · pp. 28–29
• I-O Moore transducers have both standard and time-shifted kernels, and both generate the same interface.Source: L Proof of Prop. 3 · p. 29
• For I-O Moore transducers, the output and transition factorization yields the stated belief-update rule.Source: L Proof of Prop. 3 · p. 30
• Postdictive beliefs likewise define a belief transducer that reproduces the original interface when initialized with the original state distribution.Source: L Proof of Prop. 3 · p. 30
• The proof connects these update equations to the predict and update steps of Kalman and Bayesian filtering.Source: L Proof of Prop. 3 · p. 31

N Proof of Theorem 3

• A predictive transducer’s coarse-graining is isomorphic to the memory states of the ϵ-transducer.The argument uses predictive-state sufficiency and strict concavity of entropy to show that compatible histories map deterministically to equivalence classes.Source: N Proof of Theorem 3 · p. 31
• The ϵ-transducer is not the only machine whose mixed-state presentation produces it; any transducer without redundant states does so.Source: O Comparing the reduction of general… · p. 33
• A finite non-predictive world model can correspond to an ϵ-transducer with infinitely many states and fractal belief-state structure.Source: O Comparing the reduction of general… · p. 33
• Every anticipation-free transducer has a canonical retrodictive construction that stores outcomes for all possible future action sequences and evolves deterministically.Source: P A canonical retrodictive world model · p. 33
• Reversing a transducer can violate anticipation-freeness, as illustrated by the delay channel’s acausal reverse-time behavior.Source: Q Example of a non-reversible transducer · p. 34
• Any transducer can be run backward using the full action sequence to reproduce the interface, but reversibility imposes additional structural conditions.Source: §R.3 · p. 35, §R.4 · p. 37

S Proof of Theorem 5

• The BMSM operator representation yields the stated predictive and retrodictive belief relations.Source: S Proof of Theorem 5 · p. 38
• Forward evolution applies the transducer operator for the observed action and output, followed by normalization.Source: S Proof of Theorem 5 · p. 38
• Reverse-time updating uses a modified inverse operator rather than the standard transducer evolution.Source: S Proof of Theorem 5 · p. 38
• Because not every transducer is reversible, the reverse operator need not itself describe a transducer, but it still retrodicts world-state distributions under an initial-state independence assumption.Source: S Proof of Theorem 5 · p. 38