PaperlayerView exact revision on arXiv

Source-linked AI summary

World Agent: Can Language Models Keep a World Running?

Weixing Chen, Weipeng Zhang, Nan An, Yang Liu, Liang Lin

arXiv:2609.32692v1cs.AI

TL;DR

Evaluations of generated worlds stop at creation, delivery, or single steps, leaving sustained operation unmeasured. The paper introduces a world agent task with maintenance and deduction tracks, and finds scores fall as pre-built structure is removed, with causal checking weakest. The benchmark also distinguishes local completion from failures in event organization.

  • Problem

    Existing evaluations stop at generation, delivery, or single-step transitions, leaving whether a delivered world can sustain correct operation unmeasured.

    Source: §1 · p. 1

  • Method

    The world agent task evaluates continued operation through maintenance, which grounds narrative events into explicit state, and deduction, which predicts world evolution.

    Source: §1 · p. 3, §3.1 · p. 4, §3 · p. 4

  • Results

    Across eight models, scores decline steadily as pre-built structure is removed, and causal-relation satisfaction is the weakest component for every model.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The benchmark makes sustained operation measurable and distinguishes local completion from failures in event organization.

    Source: §1 · p. 3, §6 · p. 10

    Under Naming–M1, case-difficulty correlations are weak and mostly insignificant, so readouts mix deductive ability with genre-specific priors.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

World models are moving from generating realistic frames to generating playable worlds, yet whether a delivered world can keep running is not tested anywhere. Existing evaluations stop at generation, at delivery, or at single-step transitions, and each stops at a different point along the way. Correct local state transitions or intermediate outcomes do not guarantee a correctly organized causal event flow. We propose the world agent task, which moves the evaluation point of world generation from the moment of delivery to the continued operation that follows. In this task, a model is not asked to generate a world. It is held responsible for keeping the world running, which requires coordinating events and carrying forward their consequences to constrain subsequent evolution. We instantiate the task in WorldAgent-Benchmark with two complementary tracks. In the maintenance track, the model must ground the events of a continuous narrative into correct transitions of the explicit world state while respecting causal, temporal, and concurrency constraints. In the deduction track, the model must predict how the world will evolve under partial observations and act toward a goal. The maintenance track combines LLM-assisted semantic judgments with programmatic validation and scoring, while the deduction track is evaluated entirely programmatically. Individual judgments are auditable against world states and execution logs, and scores can be recomputed from the saved judgments and execution records. Across 8 models, scores decline steadily as pre-built structure is removed from the world, and causal-relation checking is the weakest component for every model. The benchmark makes the continued operation of a world measurable and distinguishes local completion from failures in event organization. Code and dataset will be released on https://github.com/HCPLab-SYSU/WorldAgent-Benchmark.

1 INTRODUCTION

Existing evaluations of generated worlds stop at creation or delivery, leaving sustained operation unmeasured. The paper proposes the world agent task and a two-track benchmark; across eight models, scores decline as pre-built structure is removed, with causal-relation checking the weakest component.

1 INTRODUCTION

  • Prior evaluations measure visual fidelity or short rollouts for generative worlds, and coding-agent games at delivery, leaving sustained operation unmeasured.Longer interactions can degrade state consistency in generative worlds.Source: §1 · p. 1
  • Correct local outcomes do not guarantee an organized event flow: events must respect causal, temporal, duration, and concurrency constraints, with consequences propagated forward.For example, a gate may end open even if a handover is omitted, a dependency is reversed, or an event falls outside its required interval.Source: §1 · p. 2
  • A shared interface, explicit state, and execution logs let event grounding and causal, temporal, and concurrency relations be checked against explicit contracts.This supports distinguishing correct local outcomes from operations lacking prerequisites or occurring in the wrong relation to others.Source: §1 · p. 2
  • The world agent task shifts evaluation from delivery to continued operation, with maintenance grounding narrative events into state and deduction predicting how the world evolves.The model is responsible for keeping an already explicit world running rather than generating it.Source: §1 · pp. 2–3
  • Across eight models, scores declined steadily as pre-built structure was removed, and causal-relation satisfaction was the weakest component for every model.The benchmark distinguishes local completion from failures in event organization.Source: §1 · p. 3
  • WorldAgent-Benchmark evaluates maintenance and deduction across 37 cases and three construction tiers using auditable judgments and programmatic scoring.Maintenance combines program checks with LLM-assisted semantic judgments; deduction compares predictions field by field with engine evolution, without an LLM judge.Source: §1 · p. 3

2 RELATED WORK

Prior work either generates worlds, tracks or predicts explicit state, or evaluates agents while an engine maintains the world. This paper instead evaluates whether a model can keep a delivered world running against pre-run ground truth.

World Models

  • Generative world models create playable environments, while explicit-state research spans narrative tracking, state representation, single-step prediction, and one-shot symbolic world generation.Generative world models synthesize environments frame by frame Ha & Schmidhuber (2018); Brooks et al. (2024); Bruce et al. (2024); Valevski et al. (2025); Parker-Holder et al. (2024); Decart et al. (2024).Source: §2 · p. 3

Coding Agents and Game Generation

  • Agent benchmarks and persistent environments score agent achievements while a fixed engine maintains the world; game-generation benchmarks score delivered games.GameLogicBench checks gameplay mechanics in Godot projects using deterministic tick-level state assertions Che et al. (2026).Source: §2 · p. 3
  • This work assigns the model responsibility for keeping a delivered world running, rather than writing game code while an engine maintains the world.The closest story-consistency evaluation reconstructs canon from the model’s narrative and uses an LLM judge for every judgment Chen et al. (2026).Source: §2 · pp. 3–4

3 THE WORLD AGENT TASK

The world agent task evaluates whether a model can sustain an explicit world by realizing and organizing events over time. It separates maintenance of narrative-driven state from prediction of evolution under fixed dynamics.

3 THE WORLD AGENT TASK

  • The task represents a world instance, its execution trace, and a finite set of constraints on event realization, state changes, causal dependencies, ordering, duration, and concurrency.Requirements can span the trace, so earlier consequences constrain later evolution.Source: §3 · p. 4

3.1 WORLD MAINTENANCE

  • In maintenance, the model receives a continuous narrative over rounds and uses a tool interface to realize events and update explicit world state against hidden round contracts.The engine checks structural validity and atomic writes but does not supply missing events or enforce semantic prerequisites; contracts assess the full execution prefix and later operations cannot repair it.Source: §3.1 · p. 4

3.2 WORLD DEDUCTION

  • In deduction, the model predicts evolution under fixed dynamics, including autonomous processes and other actors, using disclosed inputs and permitted feedback.M1 predicts autonomous evolution without feedback; M2 couples predictions with actions and feedback, and predictions do not themselves update the world.Source: §3.2 · pp. 4–5

4 WORLDAGENT-BENCHMARK

WorldAgent-Benchmark evaluates maintenance and deduction as separate tracks that share explicit world records but use different authoring procedures and sources of ground truth. Its protocols define how cases are constructed, maintenance events and relations are assessed, and deduction predictions are checked.

4 WORLDAGENT-BENCHMARK

  • The benchmark separates maintenance and deduction, which share explicit world records but use different authoring procedures and sources of ground truth.Source: §4 · p. 5

4.1 CASE CONSTRUCTION AND EXPERIMENTAL SETTINGS

  • Maintenance cases pair a fixed narrative with event-flow contracts and difficulty-specific world data, then assess frozen snapshots and execution logs against held-out contracts.Deduction cases specify initial conditions, action and process rules, and a zero-intervention timeline; the engine computes reference rollouts for prediction queries.Source: §4.1 · pp. 5–6

4.2 THE MAINTENANCE TRACK

  • Maintenance contracts specify required events, availability and use requirements, conditional-prohibition probes, and causal, temporal, concurrency, and within-chain order relations.Source: §4.2 · p. 6
  • Across maintenance tiers, the model moves from supplied types, entities, and capabilities to constructing capabilities, then all three kinds of structure from only a map.Source: §4.2 · p. 6
  • The LLM judge binds events using successful triggers and checks probes, while programs validate bindings, derive intervals, and settle relations; event completion and relation satisfaction are scored separately.A matching final state alone is insufficient to bind an event, and relation failures do not reduce the event-completion count.Source: §4.2 · p. 6

4.3 THE DEDUCTION TRACK

  • Deduction varies whether rules or qualitative descriptions are disclosed and whether feedback is available; programs compare predictions with engine-generated references using field-level F1.A match requires both the changed field’s path and resulting value; task outcomes and background maintenance are reported separately from prediction accuracy.Source: §4.3 · p. 6
  • Saved maintenance judgments and references to world states and execution logs allow review and score recomputation, but do not ensure judgment correctness or consistency across judge reruns.Source: §4.3 · p. 7

5 EXPERIMENTS

Across both tracks, performance varies by model and difficulty: maintenance scores decline as world structure is removed, while deduction improves with closed-loop feedback but remains weak at long horizons. Causal checking is a persistent maintenance weakness, and pure deduction is sensitive to information conditions.

5 EXPERIMENTS

  • The evaluation covers eight models on both tracks, using per-round completion and relation measures for maintenance and field-level F1 readouts for deduction.Three models ran all cases; the other five were evaluated on a nine-case subset.Source: §5 · p. 7

5.1 MAIN RESULTS

  • On maintenance, Qwen 3.8 Max leads the common subset with completion 0.813 and relation 0.738; on deduction, it has the highest M2 interval average (0.589).Its M2 interval score is significantly above the three full-benchmark models.Source: §5.1 · pp. 7–8

5.2 MAINTENANCE TRACK

  • Across eight models, completion and relation scores decline across rounds and difficulty tiers, with relative completion drops reaching 34.8% for DeepSeek v4.1 Flash and 18.7% for Qwen 3.8 Max.Qwen-3.8 Flash is the exception to the broader gradient, with completion falling 10.9% and relation falling from 0.721 to 0.625.Source: §5.2 · pp. 8–9, Figure 4 · p. 8
  • Causal checking (CA) is the weakest component for every model in the extreme tier, where its cross-model mean falls from 0.683 in easy to 0.356; construction becomes the weakest hard-tier component for seven models.GLM 5.3 Flash leads hard-tier construction at 0.506; the other six models score below 0.35.Source: Figure 4 · p. 8, §5.2 · p. 9
  • Qwen 3.8 Max is the only model above the cross-model mean on every maintenance component; other models show at most one distinctive strength or flat profiles.Qwen 3.8 Max is especially above mean on CA (1.33×), TM (1.30×), and CC (1.19×).Source: Figure 3 · p. 8, §5.2 · p. 9

5.3 DEDUCTION TRACK

  • Under pure deduction, naming-condition field F1 drops to 0.03–0.15 across models, while closed-loop feedback raises interval F1 to 0.16–0.53 but leaves an open–naming gap of 7%–34%.Rule-firing F1 remains 0.28–0.50 under naming, suggesting the main collapse is in predicting concrete field values.Source: §5.3 · p. 9, Table 2 · p. 9
  • Pure-deduction M1 readouts mix deductive ability with genre-specific priors, whereas closed-loop case-difficulty correlations are strong and significant (r = 0.84–0.99).The authors characterize closed-loop interval F1 as a cleaner cross-model measure.Source: §5.3 · p. 10
  • Across the horizon, M1 naming F1 falls from 0.27 to 0.04, while M2 naming stays nearly flat from 0.43 to 0.39; 48-unit lookahead remains around 0.15.Per-step feedback largely suppresses long-horizon decay, but explicit prediction 48 time units ahead remains unreliable.Source: Figure 5 · p. 9, §5.3 · p. 10

6 CONCLUSION

The conclusion frames sustained operation as distinct from world generation and reports a reproducible difficulty gradient, while the appendix describes how maintenance and deduction cases are specified and checked.

6 CONCLUSION

  • Scores fall reproducibly as pre-built structure is removed, and causal checking is the weakest component; no model predicts the world state 48 time units ahead reliably.The benchmark is presented as a reproducible target for measuring sustained operation and identifying current gaps.Source: §6 · p. 10

A CASE INVENTORY AND CONSTRUCTION PROTOCOL

  • Maintenance cases range across genres and scales, from a three-round town crisis to Ark 001, with 287 scored events across 395 map tiles.Deduction settings include an outbreak-threatened town, a leaking space station, and an espionage network.Source: A CASE INVENTORY AND CONSTRUCTION… · p. 13
  • Maintenance specifications freeze reviewed narratives, event decompositions, structure requirements, prohibitions, and relations into a consistent set of documents and runtime inputs.Every scoring requirement must trace to the narrative or initial-world facts, and evaluation results do not rewrite a frozen case version.Source: A CASE INVENTORY AND CONSTRUCTION… · p. 13
  • Deduction cases specify world dynamics and validate them through zero-intervention replay and engine-informed scoring checks, with additional baselines and diagnostics.Answers for interventions are computed from the frozen dynamics when the corresponding trajectories are evaluated.Source: A CASE INVENTORY AND CONSTRUCTION… · p. 13

B.1 MAINTENANCE REPRESENTATION AND EXECUTION

The appendix details distinct maintenance and deduction engines, auditing and scoring procedures, and the reference targets used to evaluate predictions.

B.1 MAINTENANCE REPRESENTATION AND EXECUTION

  • The maintenance state stores geography, entity types, typed entities, and reusable capabilities; rendering is visualization, not a scored output.Entities may be held by a tile, another entity, or null, and each capability performs at most one field-level write or none.Source: §B.1 · p. 13
  • The runner provides narrative inputs and queries over authoritative state and logs, while saved records support replay against frozen snapshots.Replaying saved records does not guarantee that fresh model calls produce identical trajectories.Source: §B.1 · pp. 13–14
  • The agent, not the engine, must organize event flows: accepted trigger batches represent concurrent events, while dependent stages require separate, ordered triggers.Each batch is checked atomically against the same pre-state; rejected batches apply no proposed changes, and an accepted batch advances the discrete Step index.Source: §B.1 · p. 14

B.2 DEDUCTION DYNAMICS AND INTERACTION

  • Deduction uses a separate deterministic engine: M1 predicts without intervention, whereas M2 controls one actor while other actors and background processes continue.M1 has no true intermediate-state feedback, so Naming results partly reflect prior knowledge and guessing rather than feedback-based rule induction.Source: §B.2 · pp. 14–15
  • Deterministic deduction replay requires the same engine and case versions, initial conditions, and recorded action sequence.The run directory does not automatically preserve a fingerprinted copy of the dynamics; the corresponding source revision must be preserved separately.Source: §B.2 · p. 15

C CONTRACT ITEMS, JUDGING WORKSHEET, AND AUDIT INVARIANTS

  • Maintenance scoring combines isolated LLM judgments with deterministic checks and programs that derive event intervals, settle relations, and compute scores.An event must be bound to successful triggers whose behavior and effects satisfy its condition; a matching final state alone is insufficient.Source: C CONTRACT ITEMS, JUDGING WORKSHEET,… · p. 15
  • Event realization and relation satisfaction are scored separately, with authored causal, temporal, and concurrency requirements checked against event intervals.The causal check tests event presence and ordering against authored dependencies; it does not infer causality from temporal correlation or interventions.Source: C CONTRACT ITEMS, JUDGING WORKSHEET,… · p. 16

D DEDUCTION TARGETS, READOUTS, AND REFERENCE BASELINES

  • Deduction predictions are compared programmatically with engine-executed references, using field-level F1 for segment, interval, lookahead, and counterfactual readouts.A field match requires both its path and resulting value; interval targets include autonomous changes, not just direct effects of an action.Source: D DEDUCTION TARGETS, READOUTS, AND… · pp. 17–18

E FULL-BENCHMARK RESULTS FOR THE THREE COMPLETE MODELS

Three models ran every case on both tracks, while five others were evaluated on a discriminative subset; the study reports paired comparisons and recomputes results from saved records.

E FULL-BENCHMARK RESULTS FOR THE THREE COMPLETE MODELS

  • DeepSeek v4.1 Flash, Qwen-3.8 Flash, and GLM 5.3 Flash each ran 93 maintenance runs and 120 deduction episodes across both tracks.The deduction episodes span six cases, two modes, two information conditions, and five folds.Source: E FULL-BENCHMARK RESULTS FOR THE THREE… · p. 18
  • The other five models were evaluated on a discriminative subset of nine cases selected from the first three models’ results.The main text confines cross-model maintenance claims to the nine-case common grid, where every model ran identical cases.Source: E FULL-BENCHMARK RESULTS FOR THE THREE… · p. 18
  • All reported numbers were recomputed from saved run records, and paired contrasts used two-sided Wilcoxon signed-rank tests.Pairing was based on 93 maintenance runs, 31 cases within a tier, or the specified family-level deduction cells.Source: E FULL-BENCHMARK RESULTS FOR THE THREE… · p. 18

E.1 MAINTENANCE TRACK ON THE FULL 31-CASE GRID

On the full maintenance grid, model differences depend on score and difficulty: relation scores distinguish models more than completion, while difficulty and round progression shape both scores.

E.1 MAINTENANCE TRACK ON THE FULL 31-CASE GRID

  • Qwen-3.8 Flash leads significantly on relation in the full grid, while completion scores do not separate the three models and remain within 0.01 of their nine-case means.Its relation advantage is absent in the easy tier, appears on both scores in the extreme tier, and appears on relation in the hard tier.Source: §E.1 · p. 18, Table 5 · p. 19
  • Completion falls significantly from easy to hard for all three models; relation falls for DeepSeek v4.1 Flash and GLM 5.3 Flash but only marginally for Qwen-3.8 Flash.From hard to extreme, completion is flat, while relation continues to decay for DeepSeek and GLM but holds for Qwen.Source: §E.1 · p. 18
  • Causal checking is the weakest component for every model in every tier, while GLM 5.3 Flash’s hard-tier construction strength coincides with lower event and ordering scores.Qwen-3.8 Flash has the highest extreme-tier event score.Source: §E.1 · p. 20
  • Per-round derailment reveals failures hidden by aggregate scores: DeepSeek v4.1 Flash avoids derailment, while Qwen-3.8 Flash derails seven times and GLM 5.3 Flash twice.GLM’s Ark 001 hard run is the single full-crash run and has the lowest scores among the 279 full-grid runs.Source: Table 5 · p. 19, §E.1 · p. 20

E.2 DEDUCTION TRACK: PER-CASE BREAKDOWN

Across case families, closed-loop interval performance is more consistent across models than pure-deduction performance, while naming chiefly disrupts predicted field values rather than rule firing. Counterfactual performance varies separately from interval competence, and behavioral readouts expose differences not captured by F1 alone.

E.2 DEDUCTION TRACK: PER-CASE BREAKDOWN

  • M2-open interval F1 correlates strongly across models (Pearson r = 0.84–0.98), while M1-open correlations are weaker (r = 0.28–0.71) and Spy reverses its ranking between two models.GLM 5.3 Flash ranks Spy first and DeepSeek v4.1 Flash last under M1-open.Source: §E.2 · p. 20
  • Under naming, rule-firing F1 remains high but field F1 falls to 0.11 or below, indicating a value-level rather than rule-firing collapse.Closed-loop feedback recovers about 85% of the open–naming gap for DeepSeek v4.1 Flash, 79% for GLM 5.3 Flash, and 66% for Qwen-3.8 Flash.Source: §E.2 · p. 20
  • GLM 5.3 Flash has the lowest overall counterfactual F1 and is significantly below both other models, while DeepSeek v4.1 Flash and Qwen-3.8 Flash do not separate.Counterfactual F1 dissociates from interval competence at the family level, with near-zero cells such as Ashspring under open.Source: §E.2 · p. 20
  • All three models produced naming-condition episodes of at least 100 decision steps, while grounding rejections ranged from about 1.7 per episode for DeepSeek v4.1 Flash to about 3.1 for GLM 5.3 Flash.M1 episodes cost two to four times less than M2 episodes; GLM 5.3 Flash’s deduction batch has no token records, making its deduction readings cross-harness comparisons.Source: §E.2 · p. 21

F CONSTRUCTION OF THE NINE-CASE COMMON SUBSET

The nine-case common subset is a calibrated, discriminative selection from the 31-case library, designed to compare models while approximating full-grid results. Its selection uses case-size strata, reference-model checks, and within-stratum spread ranking, with explicit limits on representativeness.

F CONSTRUCTION OF THE NINE-CASE COMMON SUBSET

  • The nine-case grid is a calibrated subset of the 31-case library, selected before the six non-reference maintenance models ran; it covers 27 units and 129 of 444 rounds (29% of full-grid cost).A unit is one case at one difficulty tier.Source: F CONSTRUCTION OF THE NINE-CASE COMMON… · p. 21
  • Cases are stratified by checkpoint count into groups of 10, 10, and 11 because size tracks score level and how many things can go wrong better than genre.Size stratification missed reference-model full-grid means by −0.011 and −0.020 mix points, compared with −0.058 and −0.065 for family-quota stratification.Source: F CONSTRUCTION OF THE NINE-CASE COMMON… · p. 21
  • Four filters check ground-truth integrity, exclude a single degenerate reference run, require tier-order consistency, and select the three highest-spread cases per stratum.All 93 units passed the integrity check; one unit was excluded for a reference event pass rate of 0.02 against 0.84 and 0.93.Source: F CONSTRUCTION OF THE NINE-CASE COMMON… · p. 21
  • The selected nine cases span 472–1928 checkpoints and have mean spread 0.230; the subset preserves reference-model ordering and its mix means differ from full-grid means by −0.011 and −0.020.Chen 003 was excluded as saturated, while the three largest cases failed order-consistency or spread criteria; the easy tier is 0.051 below the full grid, whereas hard and extreme match within 0.01.Source: F CONSTRUCTION OF THE NINE-CASE COMMON… · p. 22

G PER-MODEL COMPONENT RADARS

Per-model component radars show normalized pass-rate profiles across six maintenance components, highlighting each model’s strongest components and differences in breadth of performance.

G PER-MODEL COMPONENT RADARS

  • Qwen 3.8 Max is the only model above the cross-model mean on all six components.Each panel averages per-component pass rates across three difficulty tiers, normalized by the cross-model mean; bold labels and filled markers identify a model’s three strongest components.Source: G PER-MODEL COMPONENT RADARS · p. 22
  • The radar profiles show single-strength patterns for GLM 5.3 Flash (A), Qwen-3.8 Flash (CA, TM), GPT 5.6 Luna (A), and Kimi K3 (CC).DeepSeek v4.1 Flash, Gemini 3.8 Flash, and GLM 5.3 have flat profiles.Source: G PER-MODEL COMPONENT RADARS · p. 22

Related papers

Loading 2609.32692v1…