PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldCast: Exploring Intelligent Visual Generation in World State Prediction

Wenhao Yan, Haodong Li, Xiaotong Li, Yaxin Luo, Zhengrong Yue, Jialv Zou, Xu Huang, Haojin Yang, Nan Jia, Yan Feng, Peng Pei, Xunliang Cai

arXiv:2610.20731v1cs.CVcs.AI

TL;DR

Higher-level visual generation tasks that integrate understanding and generation remain relatively underexplored. WorldCast benchmarks multi-step world state prediction across diverse contexts, and experiments find emerging capabilities in frontier systems alongside substantial model gaps and continuing challenges.

  • Problem

    Higher-level capabilities that tightly integrate understanding and generation remain largely unexplored, although real-world event prediction requires references to guide changes at different temporal steps.

    Source: §1 · p. 2

  • Method

    WorldCast benchmarks L3–L4 world state prediction, using heterogeneous context to guide successive visual states across 320 cases and eight categories.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    Frontier unified models handle some L3 and L4 tasks and show emerging cross-task generalization, while some domains remain challenging and model gaps are closely tied to language-side capabilities.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The analyses identify potential performance gains from interleaved generation, stronger language-model reasoning, and iterative agentic refinement for visual generation systems.

    Source: §1 · p. 3, §5 · p. 10

    Additional guidance can sometimes fail to take effect or degrade performance, and code-based simulation can take more than one hour for a run.

    Source: §4.3.3 · p. 10

Abstract

from arXiv · show

Humans interpret multimodal observations of the world and forecast how it will evolve. Recent multimodal generative systems have demonstrated similar capabilities: they can integrate multimodal context and synthesize visual outputs that depict changes in the world. To characterize this emerging trajectory, we organize model capabilities into a four-level hierarchy and discover that higher-level tasks remain relatively underexplored despite model progress. To bridge this gap, we explore model performance on multi-step world state prediction, a high-level task that requires models to forecast successive future world states from multimodal observations and diverse references. Such a task requires models to possess entangled understanding and generation capabilities, as well as extensive world knowledge to translate human action sequences, physical transformations, and abstract visual reasoning into coherent visual outputs. We introduce WorldCast, a benchmark comprising 320 manually designed cases across eight categories, and evaluate 12 representative models. The conducted experiments yield three key insights: (a) advances in frontier models are rapidly pushing the boundaries of intelligent world modeling, with the GPT-Image-2.5 System scoring 72.8 and demonstrating early-stage visual state prediction capabilities on real-world tasks; (b) WorldCast reveals substantial gaps between models: even Nano Banana Pro reaches only 42.8 and the best open-source model scores 20.1, with deficits in handling complex context, entangled references, and reasoning-heavy tasks; (c) strong generative models effectively draw on the capabilities of their underlying language models. These findings highlight the need for multimodal generative models that are capable of dynamic planning and reasoning while also being deeply unified to faithfully simulate real-world states.

1 Introduction

As multimodal generation advances, higher-level tasks that tightly integrate understanding and generation remain underexplored. WorldCast formalizes world state prediction and benchmarks models’ ability to translate heterogeneous context into successive future states.

1 Introduction

  • World state prediction requires successive visual states while interpreting what each reference means and when and how it should take effect.References can encode a state to observe, a goal, rules, patterns, object properties, or prior experience rather than only an appearance anchor.Source: §1 · p. 2
  • WorldCast targets L3–L4 tasks that translate heterogeneous context into future states, with L3 emphasizing dynamic transitions and physical plausibility and L4 requiring richer contextual integration.L4 references may encode goals, physical states, materials, experience, or patterns.Source: §1 · p. 2
  • WorldCast evaluates 12 image and video models on 320 cases across eight categories using ordered keyframe sequences and case-specific hints for sequence-level judging.The protocol treats image grids, interleaved outputs, and continuous video as ordered visual states.Source: §1 · p. 3
  • Frontier unified models show early world-simulation capability on some L3 and L4 tasks, but substantial challenges remain in simulating real-world states and advanced generative intelligence.Interleaved generation, stronger language-model reasoning, and agentic refinement suggest potential research directions.Source: §1 · p. 3
  • The paper organizes visual generative capabilities into a four-level hierarchy and identifies world state prediction as an early effort to formalize L3–L4 capabilities insufficiently explored by existing benchmarks.The task translates heterogeneous contexts into successive future world states.Source: §1 · p. 3

2 WORLDCAST

WorldCast groups world-state prediction cases by the context governing a process and the reasoning needed to render its transitions. It evaluates generated sequences for task completion, causal plausibility, and visual consistency.

2 WORLDCAST

  • WorldCast distinguishes tasks by where future-process information is encoded and how tightly reference understanding must integrate with transition generation and additional context.The benchmark comprises 320 manually designed cases and 1,352 requested visual states across eight categories.Source: §2 · p. 3, Figure 3 · p. 4
  • Given text instructions and context images, the system predicts an ordered sequence of visual world states.The sequence contains K predicted states.Source: §2 · p. 3
  • The eight categories cover L3 transitions and simulation as well as L4 tasks involving visual cues, physical properties, goals, rules, and procedural transfer.L3 also includes adaptive task completion; L4 procedural learning requires transferring an illustrated procedure to a new world.Source: §2 · pp. 3–4

2.1.1 Evaluation Dimensions

  • Task Completion measures visible realization of the intended task, while Causal Plausibility assesses whether transitions follow applicable laws, rules, or dynamics.Task Completion combines progress alignment and outcome completion with case-dependent outcome weighting.Source: §2.1.1 · pp. 4–5
  • Visual Consistency tracks persistent properties across the sequence, and the three dimensions are aggregated from 0–3 ratings into a 100-point score.Task-required changes are exempt from consistency penalties.Source: §2.1.1 · p. 5

3 Evaluation Protocol

The evaluation protocol converts model outputs into ordered states for a shared sequence-level assessment. An automatic judge scores task completion, causal plausibility, and visual consistency, with human hints supporting consistent evaluation.

3 Evaluation Protocol

  • An agentic video adapter selects task-aligned keyframes under an inclusion-first principle, but assesses required key states rather than full-video plausibility or visual quality.The adapter preserves temporal constraints while aiming to cover as many critical states as possible.Source: §3 · p. 5
  • GPT-5.5 scores progress alignment, outcome completion, causal plausibility, and visual consistency using the same protocol for image outputs and adapted video keyframes.The scores for progress alignment and outcome completion contribute to Task Completion.Source: §3 · p. 5
  • Human annotators provide judge-only hints identifying task-critical objects, milestones, constraints, and deterministic steps to improve evaluation consistency and accuracy.Source: §3 · p. 5
  • Table 1 summarizes performance for twelve image and video models across eight categories, marking within-group and overall best results and indicating generation formats.Source: Table 1 · p. 6

4 Experiments

WorldCast compares image and video systems on ordered future-state prediction, where frontier systems show early L4 capability but performance varies across tasks. Analyses find gains from language enhancement, interleaving, and agentic coding, alongside limits in complex reasoning and evaluation reliability.

4.1 Settings

  • WorldCast evaluates 12 image and video models across eight categories, scoring ordered visual states with output-specific formats and a unified multimodal-system framing.Image outputs are evaluated as interleaved sequences or cropped grid states, while video outputs use keyframes extracted by the Agentic Video Adapter.Source: §4.1 · p. 6

4.2 Main Results

  • GPT-Image-2.5 exceeds 60 on all five L4 tasks but scores 61.3 on Visual Goal Realization; model gaps generally widen on L4 and category weaknesses vary.Interleaved GPT-Image variants outperform Seedance 2.0 on key states, although Seedance scores highest among the other remaining models.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3.1 Analysis of the language model enhancer

  • GPT-6 Astra substantially improves Nano Banana Pro across L3–L4, yet it still trails GPT-Image; swapping GPT-5.5 for GPT-6 raises GPT-Image-2 only marginally.Physical Simulation declines in that swap; the results suggest effective guidance requires a strong visual generator and a closely matched enhancer.Source: §4.3.1 · p. 8, Figure 6 · p. 8

4.3.2 Analysis of interleaved generation

  • 22.7 points: switching GPT-Image-2.5 to grid output reduces overall score, with L4 Logic & Rules falling from 74.3 to 23.0.Interleaving does not explain all model gaps; as context accumulates, weaker models degrade, while GPT-6 Astra reduces Nano Banana Pro’s first-to-last-step drop from 19.0 to 6.2 points.Source: §4.3.2 · pp. 8–9, Table 2 · p. 8

4.3.3 Towards agentic visual generation with coding tools

  • 82.4 is the highest overall score under coding tools; iterative refinement raises GPT-Image-2.5 from 72.8 to 80.2 and GPT-6-enhanced Nano Banana Pro from 61.2 to 72.5.Coding helps particularly on L4 Visual Goal Realization and Logic & Rules, and L3 Physical Simulation, where it makes intermediate states explicit. Coded artifacts can guide prompts without serving as direct visual inputs; added guidance can fail or degrade output, and code-based simulation can take over an hour.Source: §4.3.3 · pp. 9–10, Table 3 · p. 9

4.4 Validity of the Agentic Video Adapter and LMM-as-a-Judge

  • Human and GPT-5.5 win rates correlate at Pearson r ≥0.97 across four dimensions, while adapter- and human-selected keyframe scores correlate at 0.9546.Six experts made blind pairwise comparisons, and three annotators selected keyframes for 100 benchmark videos.Source: §4.4 · p. 10, Figure 9 · p. 10

5 Conclusion

WorldCast frames world-state prediction as a benchmark for higher-level visual generation, where models must combine heterogeneous context with reasoning to produce coherent sequences. It reports early-stage capability in frontier unified models alongside substantial performance gaps.

5 Conclusion

  • Frontier unified models show early-stage world-state prediction capability, while substantial gaps remain and grow more pronounced for heterogeneous context and tightly coupled reasoning and generation.WorldCast evaluates L3–L4 capabilities using 320 manually designed cases across eight categories.Source: §5 · p. 10

A Related Work

  • WorldCast evaluates key visual states, whereas existing world-model benchmarks emphasize temporal continuity, scene-conditioned reactivity, or multi-turn interaction.The paper excludes interactive world models because they do not support the reasoning-based transitions and heterogeneous references required by its L3–L4 tasks.Source: A Related Work · p. 14
  • WorldCast treats language-side reasoning paired with visual generation as unified even without a unified architecture, and evaluates systems regardless of their unification approach.Its related work also distinguishes visual references for appearance, identity, spatial structure, motion, and intermediate guidance from its focus on heterogeneous references for predicting successive states.Source: A Related Work · p. 14
  • WorldCast assesses tightly coupled understanding and generation, including cases where an image specifies visual content while encoding procedural or logical constraints.Some related benchmarks approach L3 reasoning but primarily assess a static visual result.Source: A Related Work · p. 14

B Task Formulation

  • WorldCast maps interleaved image–text context and instructions to an ordered visual sequence that must satisfy visual constraints and form a temporally ordered, causally coherent process.Images may specify an initial state, goal, reference, physical property, visual rule, or intermediate demonstration; image and video models share the formulation.Source: B Task Formulation · pp. 14–15

C Data Collection

WorldCast combines existing visual materials with manually designed or synthesized scenarios, preserving the intended constraints. Its collection procedures vary by category and include single- and multi-reference cases.

C Data Collection

  • Cases combine real-world datasets, online videos, illustrated manuals, gameplay footage, and manually designed scenarios; synthetic or enhanced examples preserve intended spatial, physical, and procedural constraints.Source: C Data Collection · p. 15
  • Observation-conditioned transitions and physical simulations draw on egocentric or game data, with physical cases also sourced from science and educational videos, filtered Internet examples, and synthesized scenarios.Retained cases require grounding action logic or evolution in the observed initial state.Source: Figure 10 · p. 15
  • Visual-goal and logic-rule cases use assembly or building videos, games, and synthesized examples, while procedural-learning cases pair illustrated manuals with corresponding or analogous initial states.Visual cues include manually marked or annotated images; multi-view cases use geometrically matched home-design views enhanced while preserving 3D consistency.Source: Figure 10 · p. 15, C Data Collection · p. 16

D Evaluation Details

Human-written, case-specific hints support sequence-level evaluation by directing the judge to task-critical evidence without prescribing unnecessary appearance details.

D Evaluation Details

  • Annotators write hints after jointly inspecting each instruction, its multimodal references, and the expected progression.Source: §D.1 · p. 16
  • Hints may identify reference bindings, intermediate milestones and their order, or physical and logical constraints that must hold across states.Source: §D.1 · p. 16
  • The same hints are used across model outputs and guide Progress Alignment, Outcome Completion, Constraint Preservation, and Visual Consistency assessments without replacing visible evidence.Task Completion hints are shared by Progress Alignment and Outcome Completion.Source: §D.1 · p. 16

D.2 Evaluation Procedure

The judge reviews the full generated sequence, evaluates local transitions, and then checks global dependencies using the instructions, references, and any judge-only hints.

D.2 Evaluation Procedure

  • Evaluation uses the instruction, any judge-only hints, all reference and initial images, and the complete ordered sequence of generated images.Source: §D.2 · p. 16
  • The judge reads the full sequence, evaluates relevant local transitions, then makes a global pass to resolve dependencies that isolated frames may not reveal.Source: §D.2 · p. 16
  • Credit depends on visible evidence of state change; captions, arrows, or plausible narratives alone do not establish that a change occurred.Source: §D.2 · p. 16

D.3 Task Completion

The appendix defines task completion through ordered-sequence progress and endpoint completion, then specifies separate scales for causal plausibility and visual consistency.

D.3 Task Completion

  • TC combines normalized Progress Alignment (PA) and Outcome Completion (OC), weighting the final state by case-specific importance up to 50%.Process-focused cases set the outcome weight to zero, so TC depends solely on PA.Source: §D.3 · p. 16

D.4 Causal Plausibility

  • Causal Plausibility (CP) scores sequence-wide physical, logical, game, or task-specific rule adherence on a four-level scale from 0 to 3.Judging covers adjacent cause–effect links, long-range dependencies, prerequisites, object conservation, and temporal order.Source: §D.4 · p. 16

D.5 Visual Consistency

  • Visual Consistency (VC) scores identity and scene-attribute consistency across adjacent and non-adjacent frames on a four-level scale from 0 to 3.Task-required or causally established changes are exempt from unexplained-change penalties.Source: §D.5 · p. 17

D.6 Score Aggregation

The aggregation puts task completion, causal plausibility, and visual consistency on a shared scale before producing a weighted 100-point score.

D.6 Score Aggregation

  • TC is multiplied by 3 while CP and VC retain their 0–3 scores, then the weighted aggregate is converted to a 100-point score.The paper reports both individual metric scores and the aggregate.Source: §D.6 · p. 17
  • TC and CP each receive weight 0.4 and VC receives 0.2; these fixed category-wide weights differ from TC’s case-specific outcome weight.The aggregation gives VC half the weight of either TC or CP.Source: §D.6 · p. 17

D.7 Performance Across Evaluation Dimensions

The appendix presents representative-model scores across three evaluation dimensions and records the judge and configuration settings used in experiments.

D.7 Performance Across Evaluation Dimensions

  • Figure 11 presents representative models’ WorldCast scores across TC, CP, and VC, including GPT-Image-2.5, Sunburst, GPT-Image-2, Seedance 2.0, and Nano Banana Pro.The supplied passage identifies the figure’s scope but does not provide the plotted score values.Source: Figure 11 · p. 17, §D.7 · p. 17

D.8 Parameter Settings

  • GPT-5.5 is used as judge at medium reasoning effort; GPT-6 configurations use GPT-6 Astra at medium effort, and Codex experiments use codex-cli 0.153.4.GPT-6 Astra is used through either the Responses API or as an agent through Codex.Source: §D.8 · p. 17

E Stability of LMM-as-a-Judge

Across three LMM judges, model ordering is largely preserved despite calibration differences; repeated-generation overall-score standard deviations remain low, with more category-level variation.

E Stability of LMM-as-a-Judge

  • Model ordering is largely preserved across three LMM judges, although GLM-5.3-Flash assigns higher absolute scores and Gemini 3.1 Pro is more conservative.Figure 12 displays the overall model scores assigned by the three judges.Source: Figure 12 · p. 18, E Stability of LMM-as-a-Judge · p. 18
  • Overall-score standard deviations do not exceed 0.53 across repeated runs, while individual categories reach 2.7 points.Table 4 reports mean ± standard deviation over three GPT-5.5-scored generation runs; means cover all 320 cases and standard deviations are across generation-run means.Source: Table 4 · p. 18, E Stability of LMM-as-a-Judge · p. 18

F Additional Analysis on Coding Artifacts

This section examines coding artifacts in multimodal generation, detailing image-generation harnesses, step-level judging protocols, and criteria for evaluating visual state sequences. It also describes how coding and direct visual inputs are used across tasks and the role of multimodal guidance.

F Additional Analysis on Coding Artifacts

  • GPT-Image-2.5 improves by 20.0 points on Visual Goal Realization and 19.0 points on Physical Simulation with the coding harness over the Responses API baseline.These gains occur on tasks where the agent can explicitly represent or guide the process.Source: F Additional Analysis on Coding… · p. 18
  • Direct artifact use occurs in 22 of 35 Logic and Rules cases and 24 of 61 Physical Simulation cases, externalizing reasoning as state trajectories, geometric constructions, or process sketches.Examples include an overlay guiding GPT-Image-2.5 and a coded simulator supplying intermediate configurations for Nano Banana Pro.Source: F Additional Analysis on Coding… · pp. 18–19, Figure 13 · p. 19
  • Agents use world knowledge to adapt guidance to task needs, such as adding lines and labels to clarify subtle piston-height changes when the reference is insufficiently detailed.The authors argue future enhancers should provide task-appropriate multimodal information and visual generators should better interpret it; related approaches include CoCo (Li et al., 2026b), GenClaw (Ye et al., 2026), and Qwen-Image-Agent (Zhang et al., 2026c).Source: F Additional Analysis on Coding… · p. 19

H.3 LMM-as-a-Judge Templates

  • Endpoint accuracy receives a binary score: missing, incorrect, incomplete, contradicted, or unverifiable required final-state details earn 0, with no partial credit.Judges extract endpoint requirements from the instruction, HINTS, and references, then match each requirement to direct evidence in the final image; captions or plausible narratives alone do not count.Source: §H.3 · p. 25
  • Causal Plausibility grades whether adjacent and long-range transitions follow task-specific world rules, capping scores at 1 for unreconciled task-critical transitions.Harmless compression is not penalized when visible endpoints are mutually reachable; weakened local mechanisms support 2, unsupported critical transitions cap at 1, and impossible core mechanisms merit 0.Source: §H.3 · p. 27
  • Visual Consistency evaluates whether state that should persist remains stable across frames, exempting only changes explicitly requested or visibly required by the causal process.The audit tracks identity, count, geometry, material, topology, mechanism structure, position, camera, and background, with unexplained loss or duplication of task-critical objects warranting score 0.Source: §H.3 · pp. 28–29
  • Designated-Step Accuracy judges one paired generated image against its step-specific milestone, requiring visible realization, positive task-relevant progress, and agreement with its text plan.The evaluator uses the instruction, HINTS, references, and paired plan to identify the required change for that exact step.Source: §H.3 · pp. 29–30
  • The judge scores only the designated step, checking instruction and hint compliance, image-plan alignment, meaningful progress, physical plausibility, and object integrity using visible evidence.It returns a binary score and must not blame the step for failures elsewhere in the sequence or infer unseen changes.Source: §H.3 · p. 30

H.4 Agentic Generation Harnesses

  • The no-coding harness plans the sequence, generates images with the assigned model, inspects compliance and continuity, and selects the best sequence within shared call and time budgets.All generation and editing attempts count toward a per-case limit of 2 × step count, with a 150-minute total time limit.Source: §H.4 · p. 31
  • Both harnesses restrict generation to the assigned model and require selecting and verifying the ordered final images within shared per-case call and time limits.At the call cap, agents must stop generating, choose from existing outputs, and report any shortfall rather than bypassing the budget.Source: §H.4 · pp. 31–33
  • The coding-enabled harness requires constructing, rendering, and visually inspecting a task-specific reasoning artifact before image generation, then using its findings to refine the sequence plan.The artifact may be supplied as an image reference, but it must be meaningful and inspected; code, logs, prose, unchanged references, or post-generation artifacts do not qualify.Source: §H.4 · pp. 32–33

Related papers

Loading 2610.20731v1…