Context Compaction for World Models

1 INTRODUCTION

• Recent evaluations find that world models do not maintain good consistency when observers revisit the same area in generated worlds (Lian et al., 2025; Ye et al., 2026).A similar revisit frame can result from retrieving the original, reconstructing it from training data, or inferring it from recent frames.Source: §1 · pp. 1–2
• The benchmark separates context retrieval from context reconstruction and context inference, which prior output-based evaluations cannot distinguish.Its design addresses whether matching a revisit reflects retrieval rather than training-data recall or clues from nearby frames.Source: §1 · pp. 2–3
• The agent harness compacts a world model’s context without changing its computation or retraining its weights.The harness decides which context the underlying world model reads, analogous to context management in LLM agent harnesses.Source: §1 · p. 3
• The study varies model architecture and agent harness independently, measuring context retrieval when models read true frames and when they dream.The two layers interact: KV-cache design affects context length under fixed memory, while context compaction can extend retrieval beyond the model’s context window.Source: §1 · p. 3
• When world agents dream, models with sliding-window attention in their lower blocks lose almost all context retrieval, with or without context compaction.The experiments identify where the drop appears but do not establish why SWA lower blocks cause it.Source: Figure 1 · p. 2, §1 · p. 3

2 BACKGROUND

• A KV cache reduces decoder-only generation cost after L tokens to O(NL), while storing 2NLDρ bytes; SWA stores only 2wDρ bytes per SWA block.Without a KV cache, generation costs O(NL^2); with SWA, block compute costs O(w) with a cache and O(w^2) without one.Source: §2 · p. 3
• Decoder-Decoder models share projected keys and values across upper blocks, reducing history-dependent KV-cache storage by a factor of N relative to decoder-only models.Their cache stores 2LDρ + 2N_swDρ bytes, with SWA in the lower blocks.Source: §2 · p. 3
• Causal Encoder-Decoder models derive each decoder block’s keys and values from the final encoder output, saving prefill compute but not KV-cache size versus decoder-only models.Their prefill cost is O(N_sL^2 + N_cL), compared with O(NL^2) for decoder-only models.Source: §2 · p. 4
• Context compaction stores world history outside the model’s KV cache and selects a subset or summary when the transcript exceeds the context window.The paper frames this as a way for an external agent or program to manage the context read by a world model.Source: §2 · p. 4

3 WORLD RETRIEVAL

• The benchmark separates context retrieval from context reconstruction and context inference, enabling evaluation of retrieval apart from those capabilities.Episodes use randomly generated rooms so models cannot rely on memorized evaluation layouts; shuffled contexts test whether recent frames explain revisit predictions.Source: §3 · p. 4, §3.3 · p. 5, §3.4 · p. 5

3.1 Benchmark World

• Random walks through an 8 × 8 grid produce room revisits with gaps from a few steps to 511 steps, grouped into seven bins.Each room is a 16 × 16 tile grid, and each step is a transition through a door; the walk excludes the door just entered unless at a dead end.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 World Model

• The benchmark supports any world model that ingests its rooms and predicts the next room, while the study evaluates four 8-block designs.The models encode each frame with eight tile-type tokens and use one action token per step; all four designs have roughly 14M parameters.Source: §3.2 · pp. 4–5

3.3 Context Reconstruction

• Randomly generated evaluation rooms and separate training and evaluation seeds prevent models from reconstructing specific evaluation layouts from training data.Training on other seeds can teach general terrain statistics but not the layout of an evaluation room.Source: §3.3 · p. 5

3.4 Context Inference

• The shuffled run preserves the five most recent steps but replaces earlier history with another world, isolating retrieval beyond the local window.Five steps preserve the inputs to SWA lower blocks; context retrieval gain is the accuracy difference between original and shuffled contexts.Source: §3.4 · p. 5
• For revisits beyond 128 steps, the negative-control context retrieval gain is 0.03–0.06 across all four models, validating the isolation approach.At those gaps, neither the original nor shuffled context contains the original visit.Source: §3.4 · pp. 5–6

4 WORLD AGENTS

• The harness manages model context without retraining and is independent of the underlying model, forming a world agent together with that model.It records every frame and action in the observer’s session; compaction sends a walk through visited rooms instead of the full context.Source: §4 · p. 6, Figure 4 · p. 6

4.1 Compacted World

• The harness builds a short walk through visited rooms, retaining each room’s newest frame and ending at the observer’s current room.It plans a tour using a Christofides approximation for the traveling-salesman problem, then removes the tour’s longest edge to obtain a walk; rooms may be revisited when needed to connect the route.Source: §4.1 · pp. 6–7

4.2 Compaction Policies

• Under compact every step, the harness compacts before each model call; under compact when full, it compacts at 128 steps and then appends raw steps again.With compact when full, it compacted about once every 50 steps, and the context sent to the model held about 103 steps on average.Source: §4.2 · p. 7

5.1 Experimental Setting

• The evaluation trains on 5,000 generated 512-step episodes and tests on 400 separate episodes, averaging results over three seeds.Each evaluation samples 40 revisits from each of seven gap bins.Source: §5.1 · p. 7

5.2 Agent Harness Layer

• With context compaction, all four designs retrieve rooms up to 511 steps back—4× the 128-step window—without retraining.Raw context yields gains of [0.03, 0.06], versus [0.44, 0.54] when compacting only when full.Source: §5.2 · pp. 7–8, Figure 5 · p. 7

5.3 Model Layer

• Under a fixed 2,304-vector memory budget, DD retrieves rooms across gaps up to 127 steps, while the other designs’ gains decline beyond their shorter context windows.DD fits 128 steps; Decoder-only and CED fit 16, and CED-SWA fits 32.Source: §5.3 · p. 8, Figure 6 · p. 8
• At a fixed 128-step context, DD’s gain falls behind Decoder-only as gaps grow, reaching 0.39 versus 0.54 for gaps of 64–127 steps.The comparison attributes lower retrieval to SWA lower blocks and shared KV caches, while computing upper keys and values from z saves prefill computation.Source: §5.3 · pp. 8–9, Figure 7 · p. 8

5.4 Dreaming

• With dreaming, CED-SWA and DD reach at most 0.09 gain, while Decoder-only and CED retain gains of 0.28–0.42; the drop also appears at 8-step loops.The experiments locate the drop alongside SWA lower blocks but do not establish why those blocks cause it.Source: §5.4 · p. 9, Figure 8 · p. 9

5.5 Agent Latency

• Compact-when-full has the lowest agent latency because it prefills on only 8 of 512 session steps; its walks are replanned when context fills, about every 50 steps.Harness latency is mostly compaction time, while CED, CED-SWA, and DD prefill about 1.5 times faster than Decoder-only.Source: §5.5 · pp. 9–10, Figure 9 · p. 10

6 RELATED WORK

• Prior world-model comparisons include state recall and revisit controls, while this benchmark matches evaluated model parameter counts within 7% and uses a shuffled run to separate context retrieval from inference.R2M-Bench controls still act on model output, whereas this benchmark acts on context and generates a new world per episode to rule out context reconstruction.Source: §6 · p. 10
• Other KV-cache reductions include keeping initial and recent tokens, evicting less-important tokens, or quantizing keys and values; these techniques can in principle combine with the compared designs.The cited approaches are StreamingLLM, H2O, and KIVI.Source: §6 · p. 10
• Harness memory management is established for LLM agents, but the benchmark makes context compaction measurable by tracking whether compacted contexts preserve visited rooms.Language compaction is difficult to measure directly, while the benchmark knows which rooms were visited.Source: §6 · p. 10

7 LIMITATIONS AND CONCLUSION

• The evaluation uses one benchmark world and model scale with three seeds, and models are not trained on compacted contexts.The dreaming evaluation is limited to revisits and loops of up to 128 steps.Source: §7 · p. 10
• The benchmark’s predetermined room grid simplifies compaction; continuous camera motion would require the harness to identify frames showing the same place.Source: §7 · p. 10
• Context compaction extends retrieval beyond the context window, but dreaming preserves consistent retrieval only in models whose lower blocks attend over the full context.The conclusion says compaction is often easier than sacrificing full attention in lower blocks for sliding-window attention.Source: §7 · p. 10

AI USAGE DISCLOSURE

• Generative AI assisted with code, experiments, drafting, figures, research, and browsing; the authors reviewed and verified the work.Claude Code and Codex were also used to run experiments under author direction.Source: AI USAGE DISCLOSURE · p. 12

A RIGOROUS BACKGROUND

• Decoder-only Transformers mix information between tokens through self-attention; sliding-window attention restricts access to a bounded window.The appendix also gives full-attention forward-pass complexity as O(NL^2).Source: A RIGOROUS BACKGROUND · pp. 13–14

A.1 Inference and KV Cache

• KV caching reduces next-token generation from O(NL^2) to O(NL), while using memory proportional to history; sliding windows bound per-token work and cache size.Source: §A.1 · p. 14

A.2 Decoder-Decoder Models

• Decoder-Decoder models use lower blocks with bounded KV caches and upper blocks that read the full prefix through a shared key-value pair.The history-dependent cache component is reduced by a factor of N relative to a standard decoder-only model.Source: §A.2 · pp. 14–15

A.3 Causal Encoder-Decoder Models

• Causal Encoder-Decoder models retain the standard model’s KV-cache size but reduce prefill complexity to O(NsL^2 + NcL), with O(NL) decode complexity.Source: §A.3 · p. 15

B FURTHER EVALUATION

• The section presents exploratory deep dives across multiple aspects of the evaluation.Source: B FURTHER EVALUATION · p. 16

B.1 Agent Harness Layer

• Beyond 128 steps, raw-context retrieval gain for all four designs drops to near zero.Source: §B.1 · p. 16
• With context compaction, retrieval gain stays roughly flat across all gap bins.Source: §B.1 · p. 16
• At gaps of 8–127 steps, compaction almost always scores below raw context; the authors suspect shuffled runs can retain the revisited room among the five most recent steps.Source: §B.1 · p. 16

• Beyond 128 steps, truncating context drops retrieval gain to near zero, similar to using no context compaction.The alternative keeps only the most recent steps.Source: §B.2 · p. 16
• A shorter context alone does not explain context compaction’s improved retrieval beyond the model’s context window.Source: §B.2 · p. 16
• The comparison evaluates context compaction against truncation that retains the most recent steps.Source: §B.2 · p. 16

B.3 Dreaming

• With dreaming, CED-SWA and DD have context retrieval gain of 0.14 or less at every loop length and with either context.Without dreaming, raw-context gain for CED-SWA and DD falls as loops grow, while compacting at every step keeps every model’s gain roughly constant.Source: §B.3 · p. 16

B.4 Memory Budget

• A 2,304-vector KV-cache budget fits 128 steps in DD, 16 in Decoder-only and CED, and 32 in CED-SWA.DD stores two vectors per token; Decoder-only and CED store 16, and CED-SWA stores eight; fixed-cost SWA vectors are excluded.Source: §B.4 · p. 16
• The 2,304-vector budget is about 3.5 MB, with excluded SWA blocks adding a fixed cost of about 3.125% for CED-SWA and DD.Source: §B.4 · p. 16

B.5 Timing Methodology

• Decode latency measures predicting one step with the previous 127 steps already in the KV cache, at batch sizes 1 and 16.Source: §B.5 · p. 17
• Prefill latency measures reading a 128-step raw context into an empty KV cache and predicting the next step.For CED, CED-SWA, and DD, only the upper blocks need prefill at the last step.Source: §B.5 · p. 17
• Agent latency per step is the sum of harness and model inference latency, measured by replaying the first evaluation episode.Harness latency is the median of three context-building runs; inference latency is the median of ten decoding or prefilling runs.Source: §B.5 · p. 17