PaperlayerView exact revision on arXiv

Source-linked AI summary

From Single-Actor Environments to Multi-Actor Event Worlds: Can LLMs Be Event World Models?

Juntong Ni, Zewen Liu, Bohan Wang, Lige Zhang, Wei Jin

arXiv:2610.30488v1cs.AI

TL;DR

Many multi-actor systems evolve between an agent’s actions and are only partially observed, leaving event logs that omit parts of the state and other actors’ future decisions. The paper defines event worlds and evaluates LLMs on Diplomacy and Polymarket benchmarks. LLMs improve when all actors’ actions are known but struggle when those actions are unknown, making anticipation of other actors a central challenge.

  • Problem

    Many systems involve multiple actors, evolve without the agent acting, and are only partially observed through event logs, which leave other actors’ future actions unknown.

    Source: §1 · pp. 1–2

  • Method

    The paper formalizes event worlds and evaluates LLMs on complementary Diplomacy and Polymarket benchmarks across prediction, simulation beyond the log, and agent usefulness.

    Source: §1 · pp. 1–2, §4 · p. 4

  • Results

    LLMs learn world responses when all actors’ actions are given but struggle when only the agent’s action is known, and larger models and more data do not close this gap.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Modeling other actors’ unknown actions is a central direction for improving event world models.

    Source: §7 · p. 8

    The study covers only Diplomacy and Polymarket using prepared observations, and its agent evaluations use fixed opponent policies in Diplomacy.

    Source: Limitations · p. 9

Abstract

from arXiv · show

World models help agents assess the consequences of their actions before acting. Many real systems, however, involve multiple actors who make their own decisions, can evolve even when the agent takes no action, and are only partially observed through event logs. We formalize these systems as event worlds and ask whether large language models can predict future observations from the log and the agent's proposed actions. We build complementary benchmarks from human Diplomacy games and real Polymarket trades. Our hierarchical framework compares prediction with and without other actors' actions, then tests multi-step rollouts, predictions for actions never taken, and usefulness to agents. Fine-tuning and more data improve prediction when all actors' recorded actions are given. With only the agent's action, models struggle when other actors strongly affect the outcome, and larger models and longer histories help little. This gap also limits multi-step prediction and agent decisions, with models overestimating plan success. Yet imagined experience and initialization from world models can still improve policy learning. Anticipating other actors' actions remains a central challenge for event world modeling.

1 Introduction

Many multi-actor systems evolve between an agent’s actions and are only partially visible in event logs, making action-outcome prediction difficult. The paper defines event worlds, builds benchmarks, and evaluates LLMs, finding that predicting other actors’ unknown actions remains a central challenge.

1 Introduction

  • Event worlds involve multiple decision-making actors, can evolve without the agent acting, and are only partially observed through timestamped action-and-observation logs.An event world model predicts an observation at a chosen horizon from the log and the agent’s proposed actions, while other actors’ actions remain unknown.Source: §1 · p. 1
  • The paper asks whether LLMs can predict future observations from event logs and proposed actions, since logs omit parts of the state and leave other actors’ future actions unknown.Such predictions can help assess actions before acting, particularly when testing alternatives in the real system is costly or restricted.Source: §1 · pp. 1–2
  • LLMs predict better when every actor’s recorded actions are given, but struggle when only the agent’s action is known, especially when other actors strongly affect outcomes.Larger models do not close this gap; it also affects rollouts, alternative-action predictions, and agent decisions, where models can overestimate plan success.Source: §1 · p. 2
  • The study defines event worlds and event world models, builds two complementary benchmarks, and evaluates prediction, out-of-log simulation, and usefulness to agents.The benchmarks use human Diplomacy games and Polymarket trade logs.Source: §1 · p. 2

2 Related Work

The paper extends world-model evaluation from single-actor environments to multi-actor event logs, using human game records and real market logs to study the effect of unknown actions.

2 Related Work

  • Unlike many multi-agent benchmarks generated by LLM agents in predefined environments, these prediction tasks use human game records and real market logs.The paper also conditions event forecasting on the agent’s proposed action.Source: §2 · p. 2
  • Unlike visual world models that typically predict at regular intervals, event logs may record actions and observations at irregular times.The models take the prediction horizon as an input.Source: §2 · p. 2
  • The paper examines how other actors’ unknown actions affect prediction and agent usefulness within a unified formulation and evaluation.This extends work on LLM world models in text, web, and tool environments, event forecasting, and domain-specific market or clinical simulators.Source: §2 · p. 2

3 Problem Definition

The paper models event worlds as partially observed multi-actor systems and defines prediction tasks that distinguish known joint actions from unknown actions by other actors.

3 Problem Definition

  • An event world has multiple actors, a partially hidden state, and a log of recorded actions and observations; its state can evolve even when no action is recorded.The state includes the latest observation and hidden factors such as intentions, unrecorded actions, or internal conditions.Source: §3 · p. 3
  • An event world model predicts a future observation from the event log, the agent’s proposed actions, and a chosen time horizon.The agent may choose an empty action, meaning it waits, while other actors choose actions at the same time.Source: §3 · p. 3
  • Task S supplies the event log and only the agent’s proposed actions, whereas Task J also supplies other actors’ actions.The difference between tasks measures how much knowing those actions improves prediction.Source: §3 · p. 3

4 Event World Benchmarks

The benchmarks convert Diplomacy games and Polymarket trades into agent-perspective prediction samples, with different forms of outcome verification and market observability.

4 Event World Benchmarks

  • Both benchmarks support prediction of recorded outcomes, but only Diplomacy provides an engine to check actions never taken and evaluate agents.Polymarket tests prediction in a real market with many traders acting at any moment.Source: §4 · p. 3
  • Diplomacy provides a rules engine to determine the next board from the current board and all players’ orders, while Polymarket trades occur against an unobserved order book.In Polymarket, offers and trader beliefs can change between recorded trades, leaving prices uncertain even when orders are known.Source: §4 · pp. 3–4
  • The benchmarks yield 940k Diplomacy transitions and 1.02M one-hour Polymarket transitions, split by game or market event.Diplomacy records came from 33,279 games, retaining the 99.7% exactly reproduced by the rules engine; Polymarket data covered 4,176 markets and 79M trader orders.Source: §4 · p. 4

5 Experimental Setup

The experiments compare prompted and fine-tuned models on Diplomacy and Polymarket, using event-log inputs to predict future observations. They evaluate prediction against task-specific baselines and metrics.

Data

  • Training and test sets contain 849k and 39k transitions in Diplomacy, and 687k and 88k in Polymarket.Fine-tuning uses 20k samples per world, split evenly between Tasks J and S; shared evaluation subsets contain 1,000 Diplomacy and 1,333 Polymarket transitions unless noted otherwise.Source: §5 · p. 4

Input and Output

  • Models receive the current observation, optional prior log history, and the agent’s action; Task J also includes other actors’ actions.They predict the next Diplomacy board or Polymarket buy and sell prices, mid-price change, and uncertainty.Source: §5 · p. 4

Models

  • GPT-6-Luna and GPT-6-Sol are prompted, while Qwen3-1.7B and Qwen3-8B are fine-tuned to test whether event logs improve predictions.Prompting tests pretrained prediction ability; fine-tuning tests whether event logs improve predictions.Source: §5 · p. 4

Baselines

  • Baselines are unchanged-board and literal-order forecasts in Diplomacy, plus no-change and LightGBM forecasts in Polymarket.The Diplomacy literal-order baseline applies only the agent’s orders in Task S; Polymarket LightGBM uses hand-crafted features from all 687k training points.Source: §5 · p. 4

Metrics

  • Diplomacy uses exact match and unit-level F1, while Polymarket uses directional accuracy and CRPS for price-change distributions.Movement phases and hours with at least two other-trader orders are high-interaction; other cases are low-interaction.Source: §5 · p. 4

6 Can LLMs Be Event World Models?

The section evaluates LLMs as event world models across four increasingly demanding questions, from prediction with all actors’ actions to helping an agent act. Models perform better when all actors’ actions are known; missing actions undermine prediction, simulation, and decisions.

6.1 Can LLMs Learn the World’s Dynamics Given Every Actor’s Action?

  • With all actors’ actions given, fine-tuned Qwen3-1.7B reaches 59.7% EM in Diplomacy and 66.1% Acc in Polymarket.The Diplomacy result exceeds the best zero-shot prompted LLM; the Polymarket result is close to LightGBM trained on all 687k transitions.Source: Table 2 · p. 5, §6.1 · p. 5
  • Task J performance rises steadily with more training data, while larger models improve performance much less.Figure 3 compares scaling in training data and model size.Source: §6.1 · p. 5, Figure 3 · p. 6

6.2 Can LLMs Learn the World’s Dynamics Given Only the Agent’s Own Action?

  • Without other actors’ actions, Qwen3-8B falls from 65.1% to 20.8% EM in Diplomacy, with the largest losses when their actions determine the outcome.In Polymarket, knowing other traders’ orders helps substantially when the agent is idle but barely when its large order drives the price.Source: §6.2 · p. 6, Figure 4 · p. 6
  • More history and data help Task S unevenly, while larger models help little and Task S remains below Task J.History can help some models and hurt others; Diplomacy gains from more data occur mainly on low-interaction transitions.Source: §6.2 · p. 6, Figure 5 · p. 6

6.3 Can LLMs Simulate Situations Not in the Event Log?

  • In five-step Diplomacy rollouts, errors accumulate, especially in Task S; fine-tuning reduces accumulation in Task J, while direct 24-hour Polymarket forecasts beat recursive rollouts.Table 3 reports step-5 F1 for Diplomacy and 24-hour Acc for Polymarket.Source: Table 3 · p. 7, §6.3 · p. 7
  • For actions never taken, fine-tuned models misjudge outcomes in Task S but improve sharply when other players’ recorded orders are provided.In Task S, they predict a change in more than half of cases where the original and modified orders produce the same board; Table 4 reports F1 and spurious rates.Source: Table 4 · p. 7, §6.3 · p. 7

6.4 Can LLMs as Event World Models Help an Agent Act?

  • Plans look more successful inside LLMs than in the engine, and choosing among candidates helps mainly when joint actions are known.With B=10 candidates, Qwen3-1.7B trained on 320k transitions nearly matches the engine in Task J; in Task S, predicted gain rises while realized gain does not.Source: §6.4 · pp. 7–8, Figure 6 · p. 8, Figure 7 · p. 8
  • Imagined games can replace some real games for policy training, and initializing from a larger world model further improves the cloned policy.One thousand imagined games nearly match 1,000 real games, but 2,000 imagined games produce a weaker policy than 2,000 real games; initialization from the 320k world model outperforms initialization from the 20k model.Source: §6.4 · p. 8, Table 5 · p. 8

7 Conclusion

The paper defines event worlds and introduces Diplomacy and Polymarket benchmarks to evaluate LLMs as event world models. Models learn responses when all actors’ actions are given but struggle when only the agent’s action is known, making other actors’ unknown actions a central challenge.

7 Conclusion

  • The paper defines event worlds and introduces complementary Diplomacy and Polymarket benchmarks.The findings identify modeling other actors’ unknown actions as a central direction for improving event world models.Source: §7 · p. 8
  • Fine-tuned LLMs learn world responses given all actors’ actions, but struggle when only the agent’s action is known.More data and larger models do not close this gap.Source: §7 · p. 8
  • In Diplomacy, the limitation also affects predictions for alternative actions and agent decisions.The evaluation covers prediction, simulation beyond the log, and usefulness to agents.Source: §7 · p. 8

Limitations

The study’s findings are bounded by its two evaluated domains, prepared observations, and Diplomacy-based intervention and agent evaluations. Fixed opponent policies and observational Polymarket analysis further limit what those evaluations establish.

Limitations

  • The study covers only Diplomacy and Polymarket, using observations prepared from event logs rather than raw event streams.Broader domains and evaluation directly from raw streams are needed to assess generalization.Source: Limitations · p. 9
  • Alternative-action outcomes and agent evaluations are verified only in Diplomacy using its rules engine.Agent experiments use fixed opponent policies, so findings may not extend to humans or opponents that learn over time.Source: Limitations · p. 9
  • The Polymarket intervention analysis relies on associations in observed trades and does not establish causal effects.Source: Limitations · p. 9

Potential Risks

The paper situates event world modeling among work on logged systems, world models, and multi-agent prediction, then describes how Diplomacy and Polymarket records become benchmarks. It also cautions that unchecked predictions can overstate plan success, particularly in high-stakes settings.

Potential Risks

  • Unchecked predictions can overstate plan success when other actors’ actions are unknown, risking poor decisions in financial or medical settings.The paper recommends evaluation using only pre-action information and making prediction uncertainty clear.Source: Potential Risks · p. 9

A Extended Related Work

  • Related work spans world-model regimes, but event worlds are characterized by how they change, who acts, and how they are observed.Event logs may be irregularly timed, and the models take the prediction horizon as input.Source: A Extended Related Work · p. 12
  • Prior work models actors or their behavior within a world, while this study evaluates predictions of future observations and their usefulness to an agent.The benchmarks use human game records and real market logs, unlike many multi-agent benchmarks built from LLM-generated interactions; event forecasting predicts what happens rather than what happens after a given actor acts.Source: A Extended Related Work · p. 12
  • Unlike offline logged-data methods, which cannot identify outcomes for actions never taken without assumptions, this study compares predictions with and without other actors’ recorded actions.It presents a unified formulation across two event worlds and tests prediction usefulness to an agent.Source: A Extended Related Work · pp. 12–13

B.1 Diplomacy

  • Diplomacy observations include the phase and full board, and its rules engine deterministically resolves joint orders into unit-level results.The main experiments retain 33,194 of 33,279 no-press games whose replayed orders reproduce every recorded board; decision samples omit the map and legal orders.Source: §B.1 · pp. 13–14

B From Raw Records to Event Worlds

  • Table 7 maps the general definition to the two worlds: changed actions have engine-verified outcomes in Diplomacy, while Polymarket provides an observational response check (Appendix G.3).Source: Table 7 · p. 13

B.2 Polymarket

  • Polymarket observations summarize market context and recent trade history with price, spread, depth, activity, and agent-history features.Taker wallets are actors, while makers’ resting quotes are hidden; future prices remain uncertain even when recorded orders are supplied.Source: §B.2 · p. 14
  • Polymarket decision windows center on large orders or uniform-time controls, and the default one-hour horizon captures typical trader responses.Task J receives all window orders without execution prices, whereas Task S receives only the agent’s orders; 45% of large orders are followed by further orders from that wallet within the hour.Source: §B.2 · p. 14

B.3 Splits and Evaluation Sets

This material specifies how the two benchmarks are split and evaluated, then details the prompts, training, baselines, metrics, and scaling results. More data raises Polymarket Task S performance nearly as much as Task J, but the gap widens on high-interaction windows.

B.3 Splits and Evaluation Sets

  • Diplomacy uses 19,721 training games and 899 test games; Polymarket splits by parent event and has 2,170 training and 258 in-distribution test markets.The shared evaluation subsets contain 1,000 Diplomacy transitions and 1,333 Polymarket decision points.Source: §B.3 · pp. 14–15

C Evaluation Framework in Detail

  • Table 8 lists one experiment per row in the Section 6 evaluation framework.The split passages describe the shared test subsets used for evaluation.Source: §B.3 · p. 14, Table 8 · p. 15

B.4 Artifacts, Licenses, and Privacy

  • The Diplomacy records omit player names, while Polymarket wallet addresses are replaced with per-sample pseudonyms; the benchmarks are intended for research.The paper reports using the listed datasets and software for non-commercial research under their terms.Source: §B.4 · p. 15

D.1 Prompts and Serialization

  • Prompts serialize market state and either all orders or only the agent’s orders, while Diplomacy inputs can include prior phases; fine-tuned market outputs encode price change and future execution prices.Fine-tuned market models derive p_zero and change percentiles from 21 bins, with a separately generated point forecast.Source: §D.1 · p. 16, §D.1.1 · p. 17

D.2 Models

  • Qwen3 models are fine-tuned for three epochs, and three-seed checks on Qwen3-1.7B show score standard deviations below one percentage point.The reported experiments used about 550 GPU hours; each fine-tuned result otherwise comes from one training run.Source: §D.2 · p. 17

D.3 Baselines

  • Diplomacy baselines include literal order application and alternative order profiles; Polymarket uses no-change and LightGBM baselines, with accuracy and CRPS among its metrics.CRPS is computed from an interpolated distribution built from the predicted lower percentile, point forecast, and upper percentile.Source: §D.3 · pp. 17–18, §D.4 · p. 18

E.2 Scaling Results

  • In Polymarket, more data raises Task S nearly as much as Task J, but the high-interaction gap widens from 3 to 10 points; in Diplomacy, data gains in Task S come from low-interaction transitions.The authors note that hyperparameters are not tuned per model size, which may explain why the 8B model falls below the 4B model in Task J in both worlds.Source: §E.2 · p. 18

E.3 From-Scratch Control

From-scratch training can approach pretrained performance in some settings, but pretraining is more valuable when interactions matter and when joint actions are known. The section also examines horizon prediction, robustness, and prompted models’ claimed outcome knowledge.

E.3 From-Scratch Control

  • At 320k transitions, scratch training matches pretrained Task J on low-interaction Diplomacy but learns less on high-interaction transitions; in Polymarket it reaches pretrained 20k Task J accuracy but trails with equal data, especially on later markets.The scratch model also fails on two-player games by producing seven-player boards.Source: §E.3 · p. 18
  • Pretraining brings larger gains when joint actions are known; prediction from only the agent’s action remains difficult under both initializations.In Diplomacy Task S, the models are close at 320k, while the pretrained model is ten points ahead at 20k.Source: §E.3 · pp. 18–19

E.4 Prediction at Different Horizons

  • The horizon probe compares predicted and observed no-price-change frequencies across 5-minute, 1-hour, and 24-hour Polymarket forecasts.This passage describes the probe setup, not its findings.Source: §E.4 · p. 19

E.5 Robustness Results

  • On high-interaction transitions, Task S is no better than literal order application in either Diplomacy setting; in Polymarket it remains well below Task J in both periods.The robustness comparison uses common evaluation sets.Source: §E.5 · p. 19

E.6 Outcome Knowledge of the Prompted Models

  • GPT-6-Luna and GPT-6-Sol report knowing most test-market outcomes and are correct in 97.9% and 98.8% of those cases, respectively.Their one-hour prediction accuracy does not fall on later markets despite fewer claims of outcome knowledge; the comparison does not test when news arrived within an hour.Source: §E.6 · pp. 19–20

F.1 History-Length Sweep

Longer histories do little to resolve the action-only prediction gap, while predicting other actors’ behavior remains difficult. The section also reports limited gains from negotiation messages and shows that the value of other actors’ orders depends on whether the agent is idle.

F.1 History-Length Sweep

  • History helps Diplomacy Task S only on low-interaction transitions; high-interaction EM stays below 2% at every tested history length, and history does not close the Task S–Task J gap.In Polymarket, history changes Task S only slightly; Task J effects are also small for most models and settings.Source: §F.1 · pp. 20–21

F.2 Predicting the Other Actors’ Behavior

  • Replacing fine-tuned Diplomacy board predictions with engine outcomes helps little, while prompted models benefit more; even with three phases of history, fine-tuned high-interaction EM stays below 2%.On interacting units, fine-tuned Qwen3-1.7B beats GPT-6-Sol, which does no better than the most frequent-order baseline; DipNet has the highest per-order accuracy but only 0.1% high-interaction whole-board EM.Source: §F.2 · p. 21

F.3 Negotiation Messages

  • Adding raw negotiation messages brings little improvement: messages do not raise fine-tuned Task S EM, and the prompted model’s remaining gain is on low-interaction transitions.Messages raise the fine-tuned SF model’s Top-1 by about one point; GPT-6-Sol’s EM also rises when message text is blanked.Source: §F.3 · p. 22

F.4 Idle Agents

  • In Polymarket, other traders’ orders raise fine-tuned accuracy by about 22 points when the agent is idle but only about one point when it places a large order.In Diplomacy adjustment phases, the gain is large for idle and acting agents, where other players issue about four orders per sample versus about one from the agent when it acts.Source: §F.4 · pp. 22–23

G.1 Multi-Step Rollouts

  • On Polymarket, no model beats the no-change forecast on price error, directly or by rollout; mean absolute error is at least 1.02 cents versus 0.99 cents.GPT-6-Luna’s direct forecast predicts no change in 45.4% of Task J chains and 34.2% of Task S chains, while only 3.6% have no 24-hour mid-price change.Source: §G.1 · p. 23

G.2 Actions Never Taken in Diplomacy

The section tests counterfactual actions in Diplomacy and price responses in Polymarket, then specifies the agent-evaluation setup. Fine-tuned Diplomacy models benefit from recorded opponents’ orders when predicting alternative actions, while Polymarket price-response correlations remain weak.

G.2 Actions Never Taken in Diplomacy

  • Giving other players’ orders sharply raises fine-tuned models’ F1 and reduces spurious changes for alternative Diplomacy orders, but prompted models show no comparable gain.For prompted models, GPT-6-Sol’s F1 changes little and GPT-6-Luna’s falls; both have higher spurious rates in Task J, and repeated API calls can vary despite fixed joint actions.Source: §G.2 · pp. 23–24

G.3 Price-Response Check in Polymarket

  • On Polymarket order reversals, GPT-6 models usually predict the reference response’s sign, while fine-tuned models do so in about half of cases; predicted effects correlate weakly with relative order volume.The order-response reference is an observational curve fit to training-market averages.Source: §G.3 · p. 24

H Usefulness to Agents (Q4)

  • Agent evaluations use seven-player Diplomacy games against greedy supervised DipNet opponents; success means retaining at least the initial number of supply centers after 10 phases.Plan consistency uses 500 starting boards, and the other agent experiments share 200.Source: H Usefulness to Agents (Q4) · p. 24

H.1 Plan Consistency in Diplomacy

Plan consistency is weakest when the world model receives only the agent’s orders: it overestimates success inside the model relative to replay in the engine. The section also examines candidate selection and policy learning from real and imagined games.

H.1 Plan Consistency in Diplomacy

  • In Task S, every model’s world-model success exceeds replay success, so plans made inside the model are overestimated; Task J has lower world-model success than Real for every model.GPT-6-Luna often fails to keep an imagined Task J trajectory valid to the end, making its world-model success lower than W2R.Source: §H.1 · pp. 24–25, Table 24 · p. 25

H.1.1 Consistency with a Different Agent

  • Changing the agent from supervised to RL-trained DipNet changes CR little in either task, and Task S again has WM above W2R.Source: Table 25 · p. 25, §H.1.1 · p. 25

H.2 Choosing among Candidate Actions

  • Candidate-action selection scores proposed orders by predicted centers and surviving units, then executes the highest-valued candidate; engine scoring provides a reference baseline.The world model receives deterministic opponents’ actual orders in Task J, but only the agent’s candidate in Task S.Source: §H.2 · p. 25

H.3 Imagined Games and Initialization

  • 72.5% of imagined trajectories are judged successful by the world model, versus 62.9% in the engine.These are 10-phase games from 2,000 training-split starts, with DipNet ordering only the agent in the imagined Task S games.Source: §H.3 · p. 26
  • Behavior-cloning data use successful real or imagined games’ board-and-order pairs; initialized policies train on Real 1k from a 20k or 320k Diplomacy world model.Real 2k and Imagined 2k add successful games from 1,300 training starts disjoint from the first 2,000.Source: Table 26 · p. 25, §H.3 · p. 26
  • Real 2k gains 0.345 more supply centers than Real 1k (SE 0.10), while Imagined 2k gains 0.14 over Imagined 1k (SE 0.12).Imagined 2k is 0.25 below Real 2k (SE 0.11), paired over 200 test starts.Source: §H.3 · p. 26

Related papers

Loading 2610.30488v1…