PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Do Conventional World Models Fail to Learn Cellular Automata?

Shaoyang Guo, Ziming Liu

arXiv:2609.39604v1cs.AI

TL;DR

The paper asks whether standard world models infer exact unseen cellular-automaton rules from observed histories. It tests three information-flow changes targeting spatial locality, temporal pairing, and frame stability, finding that each repairs a corresponding failure in the reported settings.

  • Problem

    It is unclear whether standard autoregressive or diffusion world models can infer an unseen cellular automaton's rule from observed history, beyond predicting future states accurately.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates standard architectures on cellular automata and tests simple information-flow changes targeting spatial locality, temporal situation–successor binding, and temporal stability.

    Source: §1 · pp. 2–3

  • Results

    Across cellular-automaton tests, targeted changes repair distinct rollout failures; two-dimensional positions raise exact Game of Life rollouts from 39.1% to 100%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Changing information flow while holding the rest of each model fixed provides a way to test which missing link limits exact cellular-automaton rollouts.

    Source: §1 · p. 3

    Evidence is chiefly from deterministic binary automata on 8 × 8 grids; physical and video directions remain transfer tests, and continuous-dynamics results are exploratory.

    Source: §7 · p. 10

Abstract

from arXiv · show

Although conventional world models - auto-regressive or diffusion models based on transformers or convolutional networks - may learn surface statistics of world dynamics, can they learn the exact world dynamics from its observed history? Leveraging cellular automata as a simple testbed, we find the answer to be no in many cases. Conventional architectures predict most pixels correctly yet rarely complete a rollout: a CNN predicts 96.3% of cells but completes 18.9% of rollouts; a joint diffusion model completes none. We trace the gap to three failure modes of these world models - namely, they fail to exactly capture spatial locality, temporal locality or temporal stability. Simple changes repair each: (1) for spatial locality, two-dimensional rotary positions lift a transformer from 39.1% to 100% on the Game of Life; (2) for temporal locality, handing each token its cell's previous-frame neighbourhood lifts the same transformer from 25.8% to 99.9% on unseen rules; (3) for temporal stability, causal freezing lifts the same diffusion weights from 42.2% to 99.9%. None of the three changes touches the architectural backbone; each only modifies the information flow within it. We also compare joint and ordered sampling on billiards and, in an exploratory study, on a simulated Burgers equation.

1. Introduction

The introduction asks whether standard world models can infer exact cellular-automaton rules from observed histories, and finds that accurate short-term predictions often do not yield correct long rollouts. It presents three information-flow changes aimed at spatial locality, temporal situation–successor binding, and temporal stability.

1. Introduction

  • Cellular automata provide a clean testbed because simple, well-defined local rules can generate rich behavior while making the underlying rule explicit.Source: §1 · p. 1
  • 39.1% to 100% exact Game of Life rollouts: two-dimensional positions repair the transformer's spatial-locality failure without changing parameters.Source: Figure 1 · p. 2, §1 · p. 2
  • 25.8% to 99.9% unseen-rule rollouts: adding each token's cell-specific previous-frame 3×3 neighbourhood binds situations to successors.Source: Figure 1 · p. 2, §1 · p. 3
  • 42.2% to 99.9% Game of Life rollouts: causal freezing improves the same diffusion weights by settling frames in order.Source: Figure 1 · p. 2, §1 · p. 3
  • Standard architectures often predict short-term cellular-automaton states accurately yet fail to complete long-horizon rollouts, motivating evaluation of exact sequence continuation.Source: §1 · p. 2
  • Comparisons change one link while holding the rest of the model fixed, distinguishing whether information is reachable from whether training learns to use it.Source: §1 · p. 3

2. Cellular automata as a testbed

The paper uses binary cellular automata to test whether models can continue a world from observed history, evaluating both cell-level predictions and exact long rollouts. Four levels progress from executing the known Game of Life rule to predicting held-out rules and cases where the observations omit the queried situation.

2. Cellular automata as a testbed

  • Each world is a binary outer-totalistic cellular automaton on an 8 × 8 torus, with a rule mapping a cell’s state and black-neighbour count to its next state.The rule family contains 2^18 lookup-table rules, including the Game of Life.Source: §2 · p. 3
  • A rollout predicts eight frames after observing eight, and SeqAcc counts only rollouts with all 512 predicted cells correct.This strict sequence metric complements pixel-level accuracy, the fraction of predicted cells correct.Source: §2 · p. 3, Figure 3 · p. 4
  • L3 tests held-out worlds, requiring rule inference from eight observed frames, while L4 withholds the queried situation and requires a prior about the rule family.L1 tests the known Game of Life rule; L2 uses new trajectories of training-half worlds.Source: §2 · p. 3
  • At 99.9% pixel accuracy, independent errors would still spoil about two in five rollouts; the best held-out rollout rate is 64.2%.Every other model in the cited comparison completes at most 18.9% of held-out rollouts.Source: §2 · p. 4

3. Spatial locality: keep neighbours together

Flattening a grid into a raster sequence can make toroidal neighbours hard to represent reliably. Two-dimensional position information repairs this spatial-locality failure, while spatial locality alone does not solve held-out-rule prediction.

3. Spatial locality: keep neighbours together

  • 100% of Game of Life rollouts are correct with two-dimensional positions, versus 39.1% with one-dimensional rotary positions.The two-dimensional result holds with the same parameters; errors with one-dimensional positions concentrate at the wrap-around edge.Source: §3 · p. 5
  • On unseen rules, 2D rotary positions lift the d128 transformer from 82.6% to 99.3% SeqAcc, higher on every seed.The paper attributes the remaining gap between representable and reliably learned locality to reliability, not attention’s ability to represent the layout.Source: §3 · pp. 5–6
  • Spatial locality is insufficient by itself: the standard CNN executes the known Game of Life exactly yet fails most held-out rollouts.The paper identifies missing temporal locality as the remaining problem.Source: §3 · p. 6

4. Temporal locality: bind each state to its successor

Temporal locality requires binding each observed situation to its outcome so that a current query can retrieve the continuation. The section tests this with convolutional and transformer designs, then examines cases where matching the situation is not enough and a rule-family prior is needed.

4.1 Convolution: count, then look up

  • A two-frame convolution makes the CNN nearly exact at L3: learned detectors reach 99.7%, while analytic detectors are exact.The learned runs that miss do so on one of the two rarest situations, each about 0.2% of cells.Source: §4.1 · p. 6

4.2 Transformer: look-back and look-ahead pairing

  • The 3D-conv and KV-shift transformers are exact at L1–L3 with 3.8k and 2.6k parameters, respectively, while the 142k-parameter 2D-conv transformer reaches 88.8%.The first two designs explicitly pair a situation key with its outcome value; the 2D-conv design must learn that pairing.Source: §4.2 · pp. 6–7
  • Any tested memory window spanning a transition reaches at least 99.8% L3 SeqAcc, versus 87.9% and 15.6% for the single-frame (t) window.The result holds with or without learned positions and in either pairing direction.Source: §4.2 · p. 7
  • Adding each token’s cell-specific previous-frame 3×3 neighbourhood lifts the d64 transformer from 25.8% to 99.9% and d128 from 85.6% to 99.9%.The feature supplies spatial alignment and the temporal pair at once; without positional encoding, these decoders reach only 8.1–8.2%.Source: §4.2 · p. 8

4.3 Level 4: when induction needs a prior

  • At L4, where the queried situation is absent from the observations, matching alone cannot determine its outcome; models therefore use the rule-family prior through tallying or repeated attention reads.The ordinary-loss 3D-conv transformer reaches 99.5% with four reads; the KV-shift transformer reaches 99.5% with four reads and evidence-balanced loss.Source: §4.3 · p. 8

5. Temporal stability: settle each frame before it is used

Rollouts require each predicted frame to be settled before later frames use it. Causal freezing improves completion, especially for Life, while its benefit depends on the dynamics and on how many rule steps each call must compose.

5. Temporal stability: settle each frame before it is used

  • Joint denoising settles no predicted frame, while an exact-transition recurrence could extend a correct prefix one frame per round, a computation training does not discover.The standard diffusion model completes no held-out rollout and reaches 24.9% even on the Game of Life.Source: §5 · p. 9
  • With weights fixed, causal freezing raises held-out completion from 4.7% to 76.3% (49 joint calls versus 48 ordered), reaching 88.2% with one call per frame.Under the common recipe, the same swap raises completion from 4.5% to 27.9%; the direction holds across recipes, but the size differs.Source: §5 · p. 9
  • At eight calls, freezing lifts Game of Life completion from 42.2% to 99.9%, but billiards changes only from 99.2% to 100%.The passage relates the contrast to Life requiring sequential local updates, while a billiard ball's later position follows from its current position and velocity in one step.Source: §5 · p. 9
  • Increasing denoiser depth does not rescue joint generation, and freezing degrades when one call must compose several rule steps.Depth-8 freezing reaches 99.1% when displayed frames are two rule steps apart, but both samplers fail on most worlds at four steps; freezing supplies intermediate states joint training does not learn to keep.Source: §5 · p. 9
  • A structured diffusion model using a clean prefix and one noisy target frame reaches 99.4% on held-out rules with 30k parameters.Its loss is applied to that target frame only.Source: §5 · p. 9
  • The denoisers select causal order in about 98% of worlds when committing the most confident frame, while least-confident-first and random orders stay below 5%.The weights encode directional priors, so this is order selection under those priors, not discovery of causality; a plain denoiser can also commit several Life frames per call and retain 99.8%.Source: §5 · p. 9

6. What the three properties have in common

The three properties describe links needed to execute a world update: spatial, temporal, and rollout-order links. The section proposes checking whether those links are reachable and learned, then supplying missing ones through information-flow choices.

6. What the three properties have in common

  • Spatial locality brings a cell's neighbours together, temporal locality pairs each observed situation with its outcome, and temporal stability settles predictions before later ones use them.These are the three links in the paper's Figure 2 account of one-step world dynamics.Source: §6 · p. 9
  • Reachability can be checked before training, but learnability requires training; the standard models tested within budget fell short at that second test.The paired questions are whether a route exists and whether training uses it.Source: §6 · pp. 9–10
  • The proposed checklist maps needed cell/frame pairs to model access, then supplies missing position, pairing, or commitment links without changing the backbone.The authors describe it as close in spirit to algorithmic alignment and Canon layers (Allen-Zhu, 2025), and compare settling intermediate states with chain of thought in transformers (Feng et al., 2023; Merrill and Sabharwal, 2023).Source: §6 · p. 10

7. Future Work

Future work proposes testing exact dynamics in richer physical systems and testing whether the same information flow survives when cellular-automaton states are rendered as video. Early continuous-dynamics experiments suggest outcomes may depend on the dynamics.

7. Future Work

  • The evidence is chiefly from deterministic binary automata on an 8 × 8 grid; exploratory continuous-dynamics results vary by system, with causal freezing helping Burgers but not linear advection–diffusion.The physical and video directions are framed as tests of transfer, and locality and pairing results in the continuous experiments are mixed.Source: §7 · p. 10
  • Future tests will use interacting particles and continuous dynamics with known simulators to check exact discrete-state agreement or declared continuous-state tolerances over specified horizons.This is intended to distinguish executing dynamics from the visual plausibility of rendered rollouts; Appendix G.2 is described as a first step.Source: §7 · p. 10
  • A rendered automaton would add textures, lighting, and camera motion while preserving simple rules, enabling tests of latent two-frame transition inference and decoded state consistency against a matched predictor.The stated question is whether the same information flow survives perception and rendering without sacrificing visual quality.Source: §7 · p. 10

Ethics Statement

The paper reports no direct ethical risk from the models or data.

Ethics Statement

  • The study uses small synthetic cellular automata from a documented simulator and involves no human subjects, personal data, or deployed systems.The authors foresee no direct ethical risk from the models or data.Source: Ethics Statement · p. 10

Reproducibility Statement

The paper documents how results and evaluation artifacts are generated and checked, while noting that public release details remain pending release checks.

Reproducibility Statement

  • Every reported number is transcribed from registered per-run logs, and evaluation corpora are pinned to checksums for reproducibility.Training data are regenerated deterministically from (seed, index); a supplementary L4 attention-depth probe used a retrospectively preserved source snapshot but passed the same audits.Source: Reproducibility Statement · p. 10

Statement on the Use of AI Assistance

This section discloses the use and review of AI assistance, situates the study among related work, and records protocol choices and corrections to preregistered predictions.

Statement on the Use of AI Assistance

  • AI assistants contributed to methodology, design, implementation, interpretation, writing, and literature work; authors reviewed the assisted work and take responsibility for it.They did not generate synthetic data; the authors checked experimental numbers against registered logs and tested AI-assisted code.Source: Statement on the Use of AI Assistance · pp. 10–11

A. Related work

  • Related work motivates exact, prefix-conditioned continuation on held-out rules and distinguishes learnability from representational capacity or a uniquely identified attention circuit.The task filters for observed evidence that determines the continuation; reported attention patterns are consistent with situation matching and successor retrieval, but do not identify a unique causal circuit.Source: A. Related work · pp. 15–16
  • Temporal pairing devices adapt prior shifts and short convolutions to observed situation–successor pairs, while the controlled diffusion comparison tests frame ordering without claiming it is universally preferable.The paired-input results concern learnability under this task and training budget, not a new pairing operation; the denoiser weights are fixed across schedules, with 49 versus 48 network calls.Source: A. Related work · p. 16
  • The study evaluates exact sequence continuation rather than visual quality and treats pairing and frame-order findings as hypotheses for broader generators, not verified transfer.It uses strict accuracy on rule-disjoint, coverage-filtered continuations and checks that observations determine the targets.Source: A. Related work · p. 16

B. Protocol and reporting

  • Rules are split by a fixed-seed permutation of their 262,144 indices, and filtering removes trajectories that query situations absent from observed frames.Without filtering, 33% of L3 trajectories query unseen situations; the analytic 3D-conv CNN scores 0.8032 on the unfiltered corpus and 1.0000 on its covered subset, while masking loss instead is neutral (∆= 0.0001, paired).Source: B. Protocol and reporting · pp. 16–17
  • L4B tests 768 trajectories with an unseen queried situation whose centre-flipped partner is observed; L4A instead withholds one of four fixed rule rows.L4B cases were selected from 3.86M draws, and training uses a 10% dose of such trajectories.Source: B. Protocol and reporting · p. 17
  • Pixel accuracy can conceal rollout errors; reported values generally use final-step EMA weights and three-seed means, with specified exceptions.At 0.999 per pixel, independent errors would still spoil about two in five eight-frame rollouts; main-text percentages are truncated to one decimal.Source: B. Protocol and reporting · p. 17

B.1 Preregistered predictions and corrections

  • On filtered data, d128 reached 0.9263/0.7646/0.8784 on L3, while the prediction that no rung would be exact held.The expected near-zero held-out-rule performance and monotonic improvement with width did not hold; d192 and d256 each left one seed at the marginal predictor.Source: §B.1 · p. 17

C. Models and training

The models share a structured detection front end but differ in how they aggregate transitions, use supervision, and impose architectural or training scaffolds.

C. Models and training

  • Figures 7–9 map models by pipeline stage, distinguishing architectural choices from training and sampling choices; Figure 8 also identifies each attention model’s input encoding.Figure 7 traces convolutional models, Figure 8 attention models, and Figure 9 diffusion models; the listed parameter counts range from 17,994 for the 3D-conv CNN to 6,581,505 for the standard transformer at d256.Source: C. Models and training · pp. 17–18, Figure 7 · p. 18, Figure 8 · p. 18
  • The shared detectors encode each cell’s situation in 18 dimensions and each adjacent-frame transition as a 36-dimensional situation–outcome code.The situation detector uses a 3×3 circular convolution; the transition detector uses a 2×3×3 convolution, followed by small ReLU layers.Source: C. Models and training · p. 17
  • The CNN tallies transition codes, the transformer retrieves outcomes from situation-matched tokens, and structured diffusion combines a backward-looking canvas with a transition tally.The diffusion model uses a clean anchor frame and trains with a frame-wise causal-freezing pattern; the transformer’s 448 observed transitions are represented as tokens.Source: C. Models and training · p. 18
  • Most models receive structure and initialization but use ordinary losses; evidence-balanced loss is the exception, separating direct-evidence and partner-only cells.Given scaffolds include the 3D convolution, selected analytic initializations, and structured diffusion’s backward receptive field, clean anchor, and per-frame noise schedule.Source: C. Models and training · pp. 18–19
  • Transformers, the standard CNN, and standard diffusion train for 40k steps, while the 3D-conv CNN and structured diffusion train for 200k steps.The former group uses batch 512 and learning rate 10−3; the latter uses batches 128 and 2048, respectively, at 5×10−4.Source: C. Models and training · p. 19

D. Per-seed results and training budgets

The per-seed results and diagnostics qualify performance by training recipe, position encoding, temporal pairing, and evaluation setting. Across these tests, small structural changes help in some settings, but gains are not uniformly robust across unseen rules or seeds.

D. Per-seed results and training budgets

  • LifeGPT’s architecture-faithful retraining reaches 64.26 ± 26.68% L3 SeqAcc; fivefold training leaves standard CNN seeds at 0.2070/0.2065/0.2051.For the structured diffusion model, retraining from initialization at 5 × 10^-4 lifts a stuck seed from 0.9175 to 0.9805, but one or two seeds remain inexact in both fp32 and bf16.Source: D. Per-seed results and training budgets · p. 19

E.2 Rotary positions on unseen rules

  • At d128 with cell tokens on unseen rules, 2D RoPE reaches 99.3% versus 82.6%, but the experiment does not establish a robust rotary-position improvement.The preregistered contrasts were not met; at d64, rotary pair-token arms reach only 30.4% and 44.3%, against 99.9% with learned positions.Source: §E.2 · p. 21

E.3 Width ladder

  • The standard transformer’s d128 rung is best on average at 85.6% L3 SeqAcc, while no width is exact and accuracy does not rise monotonically with width.L2 validation tracks L3 in every run, indicating that escaped runs infer the rule rather than recall it.Source: §E.3 · pp. 21–22, Table 6 · p. 23

F.1 Convolution: the rare-situation tail; F.2 Which pairing the 2D-conv transformer learns

  • Every learned 3D-conv CNN run remains below 1.0, with first errors concentrated on one of two rare situations; more budget, width, or batch does not repair this pattern.The 2D-conv transformer’s attention tends to pair situations with their outcomes one frame earlier, but the evidence does not identify a unique mechanism; soft retrieval errors cluster around rare situations.Source: §F.1 · p. 22, §F.2 · p. 23

E.1 Rotary positions on the Game of Life

  • 39.1% of Game of Life rollouts complete with 1D RoPE, versus 99.4–100% for the tested 2D variants, including deliberately mis-periodic controls.The 1D model’s teacher-forced errors concentrate on border cells; the results support two-dimensional position rather than the exact period as the key distinction.Source: §E.1 · p. 20, Table 5 · p. 23

F.3 Positions and pairing across designs; F.4 Matched temporal windows

  • Removing position tables from the standard d128 transformer leaves all three seeds at the marginal predictor, and adding previous-frame neighbourhoods does not rescue it.In a separate matched-window experiment, the forward (t, t+1) window without positions exceeds 95%, but forward-versus-backward performance differences are inconclusive.Source: §F.3 · p. 24, §F.4 · p. 24

F.5 Binding across designs and training streams

  • Replacing the structured diffusion model’s 2×3×3 convolution with an index-aligned KV shift yields 49.0% mean L3 SeqAcc, versus 97.7% for the convolution retrained on the same machine.The constructor comparison is recipe-specific, not an isolated causal test; training-stream comparisons are also descriptive because precision differs.Source: §F.5 · p. 25

F.6 Level 4: evidence withheld

  • At Level 4, extra attention reads close the gap for the 3D-conv transformer under ordinary loss and for the KV-shift transformer only with evidence-balanced loss.The dose results vary by model: the 3D-conv CNN’s dose cells exceed its natural-rate cells, while the structured diffusion model scores about the same with or without the dose.Source: §F.6 · p. 26

F.7 Structured diffusion: sampler swap and matched training

Sampler swaps show that the same denoiser weights can yield reliable Life rollouts, while commitment-order and physical-system tests qualify where ordered sampling helps.

F.7 Structured diffusion: sampler swap and matched training

  • L3 SeqAcc reaches 1.0000/1.0000/0.9805 in fp32 and 1.0000/0.9927/0.9893 in the bf16 re-run; L1 reaches 1.0 on all seeds.These are results for the matched-training structured diffusion model.Source: §F.7 · pp. 26–27

F.8 Commitment order

  • The per-frame-level weights choose exactly the causal order in 98.1%/98.7% of worlds and match imposed-order accuracy, but this is order selection under existing priors, not causality discovery.The checkpoints already have a backward-looking canvas encoder, paired evidence, and a clean last-observed anchor.Source: §F.8 · p. 27

G. Physical systems

  • The appendix extends the sampler comparison to billiards and three exploratory continuous-dynamics experiments, each with an exact simulator.The continuous experiments are described as exploratory.Source: G. Physical systems · p. 27

G.1 Billiards and the Game of Life

  • At B = 8, causal freezing reaches 99.9% on Life versus 42.2% with joint denoising, while free billiards reach 100% versus 99.2%.Across B ∈{8, 16, 24, 40, 49}, freezing stays at 99.8–99.9% on Life; joint denoising reaches 42.2–43.2%.Source: §G.1 · p. 28
  • On unseen rules, ordering does not supply the missing pairing: at depth 4, joint denoising scores 0.0000 on every seed versus 0.1445/0.1519/0.1587 with freezing.At depth 8, joint scores are 0.1577/0.1655/0.1616 and freezing scores 0.1411/0.0781/0.1240.Source: §G.1 · p. 28
  • Confidence-first matched causal accuracy on all six checkpoints, but ties dominated confidence rankings; parallel commitment reduced billiards accuracy by more than two points on every seed.On non-static Life worlds, parallel commitment reached 99.8% in 1.91 calls; the results support neither a universal one-frame-at-a-time requirement nor automatic dynamics-matched parallelism.Source: §G.1 · pp. 28–29

G.2 Exploratory experiments on continuous dynamics

Exploratory continuous tests find that sampler and locality outcomes vary by dynamics; proposed extensions to video, language models, and interpretability remain hypotheses, not results.

G.2 Exploratory experiments on continuous dynamics

  • The continuous experiments test interpolation within a known equation family, not a new law; two spring damping values lie just above the training range.Each experiment uses its own normalized error, so errors should be compared within experiments, not across experiments or with SeqAcc.Source: §G.2 · p. 29
  • Joint denoising is better on advection–diffusion and causal freezing on Burgers across all three seeds each; the sampler gain is not attributed to commitment alone.The samplers differ in the noise each frame sees, and why each family prefers its sampler was not tested.Source: §G.2 · p. 30
  • The permuted spring ring underperforms the true-neighbour ring on every seed; for particles, true-radius messages beat all-pairs messages on every seed by about 4%.All-pairs messages are comparable to the true ring in the spring experiments.Source: §G.2 · p. 30
  • Lower mean error did not ensure reliable rollouts: 84%, 87%, and 41% of linear-spring rollouts stayed within 5% at every predicted frame, and no particle rollout did.The experiments supplied the graph or interaction radius, and a simulator using mean training coefficients was comparable or more accurate.Source: §G.2 · p. 30

H. Extended outlook

  • The paper frames its extended-outlook directions as hypotheses from automaton experiments, not findings.The proposed extensions include transfer to physical video, diffusion language models, real-video audits, and interpretability.Source: H. Extended outlook · pp. 30–31
  • For physical video, the proposed tests include windows long enough for the dynamics’ order and motion-compensated alignment for index shifts.The text also proposes testing generation schedules against learned dependencies and first checking whether encoded states keep cells at consistent latent positions.Source: H. Extended outlook · p. 30
  • For diffusion language models, the proposed test compares joint refinement, sequential commitment, and partial commitment at matched call budgets, measuring exact execution.The suggested setting is generated tokens that carry computational dependencies, such as program traces or mathematical derivations.Source: H. Extended outlook · p. 30
  • A real-video audit would use worlds with written-down rules and strict exact-execution metrics, including tests of context-window identifiability and tolerance over prediction horizon.The outlook also proposes scoring interpretability features against known automaton quantities such as neighbour counts and rule bits.Source: H. Extended outlook · p. 31
Loading 2609.39604v1…