PaperlayerView exact revision on arXiv

Source-linked AI summary

Building Social World Models with Large Language Models

Haofei Yu, Yining Zhao, Guanyu Lin, Jiaxuan You

arXiv:2606.11482v1cs.SIcs.CL

TL;DR

Predicting how social beliefs change after major events remains difficult because the responsible events are latent and traditional opinion data can be noisy or unrepresentative. SWM models belief transitions using prediction-market states and latent event attribution, achieving state-of-the-art performance on Kalshi and competitive performance on Polymarket.

  • Problem

    Modeling social-belief shifts after societal events is difficult because the responsible events are latent and conventional opinion sources can be noisy, biased, or unrepresentative.

  • Method

    SWM models belief dynamics as event-conditioned state transitions over prediction-market trajectories, using posterior-guided latent event attribution without explicit human annotations.

  • Results

    SWM achieves state-of-the-art results on Kalshi, including a 4% Directional Accuracy improvement over GPT-5.5, and competitive performance on Polymarket.

  • Takeaways & Limitations

    The findings support further research on parametric social world models for representing collective belief evolution.

  • Takeaways & Limitations

    Simulation errors in a weak-yen corporate-profit scenario reveal that SWM can misrepresent complex macroeconomic interactions.

Abstract

from arXiv · show

Understanding and predicting how social beliefs evolve in response to events -- from policy changes to scientific breakthroughs -- remains a fundamental challenge in social science. Given LLMs' commonsense knowledge and social intelligence, we ask: Can LLMs model the dynamics of social beliefs following social events? In this work, we introduce the concept of the Social World Model (SWM), a general framework designed to capture how social beliefs evolve in response to major events. SWM learns state-transition functions for social beliefs by mining temporal patterns in social data and optimizing the evidence lower bound, without the need for explicit human annotations linking events to belief shifts, or for expensive census data. To evaluate SWM, we introduce a benchmark, SWM-bench, derived from real-world prediction markets, specifically Kalshi and Polymarket. SWM-bench includes over 12k data points for social belief prediction tasks spanning diverse domains such as politics, finance, and cryptocurrency. Our experimental results show that SWM significantly outperforms time-series foundation models, achieving state-of-the-art results on Kalshi data and demonstrating competitive performance on Polymarket data, while offering interpretable insights into the underlying mechanisms of social belief dynamics.

1. Introduction

The paper asks whether LLMs can model how social beliefs evolve in response to events, addressing challenges in quantifying and tracking these beliefs. It introduces the Social World Model (SWM) and evaluates it with SWM-Bench built from real-world prediction-market data.

  • Motivation: Social beliefs vary across communities, with some remaining stable while others shift dramatically after societal events.Examples include beliefs about AGI emergence and the next U.S. president.
  • Challenges: Modeling social-belief dynamics is difficult because beliefs are semantic-driven and high-fidelity time-series data are scarce.The introduction identifies quantifiability and data scarcity as a primary challenge.
  • Approach: SWM represents collective beliefs as semantic states and social events as exogenous inputs in a state-transition framework, P(s_t+1 | s_t, e_t).It is designed to predict future belief states from historical states and hypothetical events.
  • Approach: LLMs serve as SWM’s transition engine, supplying commonsense reasoning and social knowledge to simulate belief shifts in complex contexts.A posterior-guided mechanism is introduced to address the attribution gap.
  • Evaluation: 4% improvement in Directional Accuracy over baselines such as GPT-5.5 is reported on Kalshi, alongside competitive performance on Polymarket.These results come from empirical evaluations on SWM-Bench, a benchmark curated from real-world prediction-market data.

2. Related Works

Related work spans social event forecasting and social simulation. Event forecasting uses semantic, time-series, and increasingly LLM-based methods, while social simulation models collective reactions through LLM-driven agents and emergent macro-patterns.

  • Social event forecasting: Traditional event prediction forecasts future occurrences from historical data using semantic and time-series modeling.
  • Social event forecasting: LLMs have emerged as effective event forecasters due to their strong social reasoning, motivating new benchmarks for LLM agents.
  • Social simulation: Social simulation commonly uses LLM-driven agent-based modeling to derive emergent macro-patterns from individual micro-decisions.
  • Social simulation: Existing social simulation work often emphasizes theory-of-mind modeling or resource distribution.

3. Preliminaries

The preliminaries define prediction-market data as a foundation for tracking social belief dynamics and formalize beliefs, events, states, and candidate event spaces. Social beliefs are daily market-implied probabilities for binary propositions, while states retain recent trajectories and transitions condition on realized event candidates.

  • Data selection for social state description: Prediction markets provide a foundation for modeling dynamic social-belief shifts amid limitations of surveys and social-media data.Surveys face response and participation biases, while social-media corpora reflect self-selected populations that may diverge from the broader public.
  • Social belief: A social belief is a binary proposition q paired with its market-implied Yes probability v_t ∈ [0, 1], sampled daily from the closing price.The proposition resolves by a future time T, with the belief defined at times t < T.
  • Social event: A social event is a news-reported real-world occurrence that may drive belief evolution, with null events representing days without significant external shocks.Valid non-null events are represented as tuples, while the null event is denoted equivalently as e_0.
  • State space: The social state s_t is an ordered window of the most recent k + 1 daily belief values, allowing modeling of temporal momentum and volatility.Formally, s_t = (q, (v_{t−k}, …, v_{t−1}, v_t)) ∈ S, where S = Q × [0, 1]^{k+1}.
  • Event space: For each transition from day t to t + 1, the realized event space contains multiple candidate events, but each forward pass conditions on one event.The candidate set functions as the externally supplied action space in the world-model formulation.

4. Proposed Concept: Social World Model

The Social World Model represents social beliefs as trajectories that evolve through event-conditioned state transitions, treating real-world events as explicit drivers rather than relying only on historical trends. It supports observed and hypothetical belief-dynamics simulations through LLM-based attribution and posterior-guided training.

  • Definition and formulation: SWM models how a social belief state s_t evolves in response to a social event e_t through an event-conditioned transition distribution.The model parameters are denoted by θ.
  • Definition and formulation: Unlike standard time-series forecasting, SWM explicitly models discrete social events as drivers of belief-state transitions.This captures temporal relationships between news-reported occurrences and shifts in public opinion, enabling observed and hypothetical simulations.
  • Training framework: Training collects observational state-event transitions, aligns the social attributor Pη with posterior Qϕ using KL divergence, and trains SWM Pθ on attributed events.Only Pη and Pθ are updated during training.
  • Implementation: Because q is fixed within a trajectory, implementation predicts v_t+1 and deterministically updates the state window as s_t+1 = (q, (v_t+1−k, . . . , v_t+1)).This provides a practical next-belief-value formulation for the trajectory state.
  • Comparison to existing world-model definitions: SWM treats states as belief trajectories, events as exogenous shocks, and propositions or markets as instances sharing a universal transition function Pθ.A designated null event represents periods without observed external drivers, while shared parameters support generalization across domains.

5. Building a Social World Model with LLMs

SWM models social-belief transitions as latent single-event attribution, using a hindsight LLM posterior to train separate event-attribution and transition-dynamics models. Its design includes a null-event persistence branch and supports both uncertainty-marginalized forecasting and hypothetical-event simulation.

  • Latent event attribution: SWM trains on observational belief transitions (s_t, E_t, s_t+1) while treating the event driving each shift as latent.Learning is framed as latent event attribution because the responsible social event is often entangled in complex information streams.
  • Modeling assumptions: The model approximates each transition with one hypothesized event and assumes candidate events are conditionally exogenous given the current belief state.The single-event approximation yields a soft distribution over candidate explanations; when exogeneity fails, predictions are not causally identified.
  • Null event: The null event represents the persistence forecast s_t, supervises only the attributor, and leaves the transition model focused on non-null event dynamics.Its fixed expected next state is s_t, and under exogeneity the predicted change relative to this baseline represents an event’s abnormal effect.
  • Training objective: A frozen hindsight posterior Q_ϕ(Z_t | s_t, s_t+1, E_t) supplies sharply peaked event attributions that supervise the world model and distill into the forward attributor.The objective is posterior-guided distillation rather than variational inference because the variational posterior is frozen and the ELBO gap cannot shrink.
  • Downstream use: Decoupling attribution from transition dynamics enables forecasting by marginalizing event uncertainty and simulation by conditioning directly on a specific or hypothetical event.Forecasts combine the persistence baseline with an attributor-weighted expected move, while simulation serves as the primary interventional deployment mode.

6. Experimental Settings

The experiments evaluate news-driven belief forecasting on volatility-filtered Polymarket and Kalshi data, using controlled event timing, multiple forecasting baselines, and four complementary metrics. Additional evaluations examine performance on attributed examples and the effect of social world model and attributor size.

  • Data settings: SWM-Bench uses Polymarket and Kalshi historical data from December 2022 to January 2026, filtered for high price volatility and keyword-matched news.Candidate news is drawn from a three-day window strictly before the target observation, preventing future information from being visible.
  • Baseline settings: The comparison includes time-series-only baselines trained on SWM-Bench and prompting-based LLMs that process numerical history with available news.Listed time-series baselines include Autoformer, DLinear, Informer, iTransformer, PatchTST, TimeMixer, and TimesNet.
  • Evaluation settings: Evaluation uses a 16-day look-back window, at most 30 candidate events, and MASE, MAE, 3-way DA, and Pearson Corr.SWM forecasts through joint inference over the prior attributor and world model.
  • Evaluation settings: Results are reported separately for full and attributed test sets, with the latter containing examples where Qϕ with Qwen3.5-235B assigns non-zero probability to at least one candidate event.The four reported metrics are MASE, MAE, directional accuracy, and Pearson correlation on both Polymarket and Kalshi.
  • Ablations: Ablations on Qwen3-0.6B, 4B, and 8B show that larger social world models and larger social attributors have better performance.The world-model ablation covers Kalshi and Polymarket; the attributor ablation trains with KL and tests on both platforms.

7. Experimental Results

SWM achieves the strongest directional accuracy on both markets and performs especially well on Kalshi, while Polymarket’s structural differences reduce the benefit of event attribution for magnitude metrics. An oracle attribution analysis shows that identifying the correct causal event remains the main bottleneck.

  • Overall performance: 0.685 and 0.845 are SWM (prior)’s directional accuracies on Polymarket and Kalshi, respectively, the highest among all baselines.On Kalshi, SWM also achieves MASE 0.800, MAE 0.084, and Corr 0.380 on attributed data.
  • Market dynamics difference: SWM surpasses GPT-5.5 and all time-series models on nearly every Kalshi metric but trails massive LLMs in Polymarket magnitude metrics.SWM retains the best directional accuracy on Polymarket.
  • Market dynamics difference: 19% non-null attribution and 17% attribution for large Kalshi moves indicate learnable event updates, whereas Polymarket’s 10% attribution and 42% revert rates weaken attribution.The degraded event signal hurts attribution-based training more than purely time-series methods.
  • Attribution bottleneck: 0.894 Kalshi DA and 0.525 Corr are achieved when the hindsight posterior replaces the prior attributor, substantially improving performance.This indicates that Pθ simulates transitions accurately once the driving event is identified, making Pη’s causal-shock isolation the critical bottleneck.

8. Ablation Studies

Ablations show that larger world-model backbones improve prediction, while brief price histories and small event pools capture most available gains. Posterior selection exposes a precision–coverage trade-off in attribution quality.

  • Social world model size: Larger Qwen3 world-model backbones consistently improve prediction on both markets, reducing MASE and increasing Corr.On Kalshi, MASE falls from 0.884 to 0.738 and Corr rises from 0.290 to 0.525; on Polymarket, MASE falls from 0.977 to 0.892 and Corr rises from 0.161 to 0.439.
  • Time-series window size: Increasing the historical price window from w=1 to w=16 yields modest gains, with most improvement realized by w=4.On Kalshi’s attributed subset, MASE changes from 0.787 to 0.738 and Corr from 0.430 to 0.525; a brief history suffices for one-step prediction.
  • Event set size: Posterior-guided event selection matches full-set performance with m=1, while prior-guided selection reaches the same level by m=5.Random selection starts substantially weaker and requires the full event pool to catch up, consistent with attribution sparsity.
  • Posterior attributor model size: A sharper Qwen3.5-397B posterior has top-1 mass 0.787 but attributes external news for only 11.9% of transitions.The flatter Qwen3-32B posterior has top-1 mass 0.603 and attributes 47.8% of transitions, illustrating a precision–coverage trade-off.

9. Case Studies

The case studies show that SWM can align news with belief shifts in forecasting and simulation, while posterior attribution remains an explanatory filter rather than formal causal certification. They also demonstrate that attribution may capture spurious topical correlations.

  • Attribution limitations: Posterior attribution surfaces news using semantic relevance, temporal precedence, and directional consistency, but does not formally certify causation and can latch onto spurious correlations.
  • Causally aligned case: 0.95 attribution identifies a BoJ rate-increase article published before a 0.690 →0.844 upward belief adjustment, plausibly reflecting the genuine information shock.
  • Spurious case: 0.75 attribution assigns an Ethereum article reporting flat trading to a 0.320 →0.225 drop that later reverts, illustrating topical rather than causal attribution.The article is repeatedly reused as the cause across Ethereum threshold contracts despite differing shift directions.
  • Forecasting: 0.583 prior attribution selects Russia–Ukraine peace-deal news, which the world model maps to a positive directional shift in the Trump–Putin forecasting scenario.SWM uses the prior attributor on the candidate news pool while jointly forecasting belief trajectory steps.
  • Simulation: ∆p =+0.14 and ∆p =+0.10 are predicted for Ueda’s rate-hike signal and absent ministerial pushback, respectively, under simulation mode.Simulation bypasses the prior attributor and uses the event-conditioned world model as a pure interventional simulator.
  • Simulation: ∆p = −0.16 follows a hypothetical 30% global oil-price crash, which the simulator predicts will reduce hike probability through anticipated deflationary effects.

10. Conclusion

The paper introduces an LLM-based training recipe for social world models of collective belief evolution and evaluates it on the SWM-Bench prediction-market benchmark. SWM uses human-annotation-free, posterior-guided training to isolate latent causal drivers, achieving state-of-the-art performance on Kalshi and competitive results on Polymarket.

  • Contributions: SWM introduces an LLM-based training recipe for modeling collective belief evolution, evaluated on the SWM-Bench prediction-market benchmark.The framework is designed to model social world dynamics without requiring explicit human annotations.
  • Training paradigm: SWM uses a human-annotation-free, posterior-guided training paradigm that isolates latent causal drivers.This training approach is presented as the mechanism enabling social world modeling without human annotations.
  • Evaluation: SWM achieves state-of-the-art performance on Kalshi and competitive results on Polymarket.These results are reported on the SWM-Bench prediction-market evaluation.

Impact Statement · B. Asset Details

SWM and SWM-Bench are presented as scalable tools for modeling collective belief changes through prediction markets and LLM reasoning, with potential uses in policy and research. The paper also emphasizes risks of malicious deployment and recommends transparent, defensive use under rigorous ethical oversight.

  • Impact Statement: SWM and SWM-Bench predict how collective beliefs evolve in response to global events.The framework and dataset ground social dynamics in prediction markets.
  • Impact Statement: By leveraging LLM reasoning and prediction markets, the framework offers a scalable toolkit for quantitative digital sociology.
  • Impact Statement: SWM could assist policymakers, NGOs, and economists in anticipating changes in collective beliefs.
  • Impact Statement: Malicious deployment could weaponize SWM to optimize targeted disinformation and persuasive social engineering campaigns.
  • Impact Statement: Because SWM uses prediction markets as a proxy for public sentiment, live deployment risks enabling manipulation of public opinion.
  • Impact Statement: The authors advocate transparent, defensive deployment to detect unnatural, abrupt, or inorganic shifts in collective belief.Such detection could help researchers identify and counter coordinated influence operations.
  • Impact Statement: Applications of social world models should be governed by rigorous ethical oversight.The paper presents SWM as a powerful lens into the collective human psyche whose use requires safeguards.

B.1. Code and Data Open-source … D. Detailed Prompts

The appendix documents open-source licensing, dataset construction, model and compute configurations, training and inference procedures, evaluation metrics, and the prompts used for market forecasting and causal attribution. Together, these details specify how SWM data are collected, models are trained and deployed, predictions are evaluated, and news candidates are scored.

  • B.1. Code and Data Open-source: The dataset is released under CC BY-NC-ND 4.0, while the code is released under the MIT License, with platform Terms of Service compliance and attribution.The data derive from publicly accessible Polymarket and Kalshi information.
  • B.2. Dataset Details: News data are collected daily from December 2022 through January 2026 using GNews, with more than 10 significant headlines and descriptions per day.Candidate events use market-question keywords, a three-day pre-observation window, at most 100 results, and 30 randomly sampled candidates for training and inference.
  • B.2. Dataset Details: Prediction-market data come from publicly accessible Polymarket and Kalshi APIs and include market metadata and historical time-series data.Z-score sub-sampling removes near-flat transitions and retains statistically significant moves so the benchmark concentrates on news-driven price changes.
  • B.3. Model License: The listed Qwen3 models use Apache 2.0 licensing, GPT-5.5 is proprietary, and the world-model regression head is kept in fp32 while training uses bf16.Both components fully fine-tune all parameters under FSDP full-shard with activation/gradient checkpointing.
  • C.1. Compute Resources: Training uses 8 A100 80GB GPUs, while inference runs on a single A100 80GB GPU.This configuration supports the reported training and deployment setup.
  • C.3. Inference Details: At inference, the prior scores candidate news and a null category, the world model predicts event-conditioned shifts, and the forecast adds their attribution-weighted sum to the current state.Posterior oracle mode replaces prior weights with hindsight weights π_t; the attribution normalization follows the Bradley–Terry/Luce construction.
  • C.4. Evaluation Details: Evaluation reports MAE, MASE, three-way directional accuracy, and Pearson correlation, with MAE/MASE lower-is-better and DA/Corr higher-is-better over specified evaluated or moved subsets.The detailed prompts provide market context, price history, news, and target date; posterior attribution checks direction, counterfactual necessity, and mechanism, while most candidates should receive a low score.
Loading 2606.11482v1…