PaperlayerView exact revision on arXiv

Source-linked AI summary

The Trinity of Consistency as a Defining Principle for General World Models

Jingxuan Wei, Siyuan Li, Yuhang Xu, Zheng Sun, Junjie Jiang, Hexuan Jin, Caijun Jia, Honghao He, Xinglong Xu, Xi bai, Chang Yu, Yumou Liu, Junnan Zhu, Xuanhe Zhou, Jintao Chen, Xiaobin Hu, Shancheng Pang, Bihui Yu, Ran He, Zhen Lei, Stan Z. Li, Conghui He, Shuicheng Yan, Cheng Tan

arXiv:2602.23152v1cs.AI

TL;DR

Building a general world model requires coherent physical simulation, yet specialized capabilities may lack a shared cognitive substrate. This paper frames the challenge through modal, spatial, and temporal consistency and evaluates models with CoW-Bench, whose results show closed-source image generators leading overall while temporal rule-grounded performance remains uneven.

  • Problem

    A coherent world simulator requires a shared cognitive substrate, but current approaches often treat semantic, geometric, and causal capabilities as independent objectives.

    Source: §2.5 · p. 28

  • Method

    The paper proposes modal, spatial, and temporal consistency as core constraints and introduces CoW-Bench, which evaluates six task categories across 18 subtasks.

    Source: §1 · p. 4, §4.2.4 · p. 65

  • Results

    CoW-Bench rankings show closed-source image generation models leading overall while open-source video generators lag on consistency-sensitive tasks, and temporal rule-grounded progression varies across models despite high visual continuity.

    Source: §5.4 · pp. 72–73

  • Takeaways & Limitations

    The paper concludes that physical understanding is better characterized by robustness across cross-dimensional interactions than by single-axis performance.

    Source: §6 · p. 84

    Current benchmarks can misstate physical fidelity because MLLM judges poorly perceive fine-grained attributes and may reward smooth videos that violate physical laws.

    Source: §4.2.4 · p. 64

Abstract

from arXiv · show

The construction of World Models capable of learning, simulating, and reasoning about objective physical laws constitutes a foundational challenge in the pursuit of Artificial General Intelligence. Recent advancements represented by video generation models like Sora have demonstrated the potential of data-driven scaling laws to approximate physical dynamics, while the emerging Unified Multimodal Model (UMM) offers a promising architectural paradigm for integrating perception, language, and reasoning. Despite these advances, the field still lacks a principled theoretical framework that defines the essential properties requisite for a General World Model. In this paper, we propose that a World Model must be grounded in the Trinity of Consistency: Modal Consistency as the semantic interface, Spatial Consistency as the geometric basis, and Temporal Consistency as the causal engine. Through this tripartite lens, we systematically review the evolution of multimodal learning, revealing a trajectory from loosely coupled specialized modules toward unified architectures that enable the synergistic emergence of internal world simulators. To complement this conceptual framework, we introduce CoW-Bench, a benchmark centered on multi-frame reasoning and generation scenarios. CoW-Bench evaluates both video generation models and UMMs under a unified evaluation protocol. Our work establishes a principled pathway toward general world models, clarifying both the limitations of current systems and the architectural requirements for future progress.

1 Introduction

The introduction argues that a general world model must simulate physical reality consistently across modalities, space, and time. It proposes the Trinity of Consistency as a framework and CoW-Bench as a way to evaluate it.

1 Introduction

  • A general world model should move beyond passive observation to simulate physical laws, reason about counterfactuals, and predict future states from actions.The introduction frames this capability as central to AGI and describes the shift as becoming a proactive simulator.Source: §1 · p. 4
  • Sora and Gen-3 approximate complex dynamics, while UMMs integrate sensory inputs, yet models can exhibit structural hallucinations, temporal inconsistency, and causal violations.The introduction attributes this gap to models mimicking pixel statistics rather than internalizing physical principles, and notes the absence of a principled framework for defining a General World Model.Source: §1 · p. 4
  • The proposed Trinity requires modal alignment in a shared semantic space, spatially grounded 3D geometry with occlusion and object permanence, and temporal adherence to physical laws and causal logic.These are presented as orthogonal yet synergistic constraints on a valid internal simulator.Source: §1 · p. 4
  • The paper traces generative models’ evolution from loosely coupled specialized modules toward unified architectures, arguing that modality, space, and time must synergize for coherent world simulation.Its review covers the dimensions’ independent development, their integration in UMMs, and remaining gaps between probabilistic generators and physical simulators.Source: §1 · p. 4
  • CoW-Bench evaluates whether models maintain the Trinity under complex, open-ended multi-frame reasoning and constraint-satisfaction scenarios.The benchmark is introduced as a unified evaluation suite that tests understanding of the world rather than visual plausibility alone.Source: §1 · p. 4

2 Foundational Exploration of Consistencies · 2.1 The Anatomy of General World Models

General world models integrate modal, spatial, and temporal consistency as an information interface, geometric cornerstone, and dynamic engine. These dimensions mutually constrain one another and developed through specialized models before enabling unified world simulators.

2.1 The Anatomy of General World Models

  • Modal, spatial, and temporal consistency serve as the information interface, geometric cornerstone, and dynamic engine of world models.Source: §2.1 · p. 5
  • Modality alignment yields a unified representation space whose semantic priors support spatial geometry reconstruction.Source: §2.1 · p. 5
  • The 3D manifold of spatial consistency imposes physical constraints on temporal evolution.Source: §2.1 · p. 5
  • Specialized models first addressed modality alignment, spatial consistency, and temporal consistency as distinct challenges.Source: §2.1 · p. 5
  • Modality alignment matured through high-dimensional manifold mapping, while spatial consistency shifted from 2D proxies to explicit 3D primitives.Source: §2.1 · p. 5
  • Temporal consistency progressed from simple frame interpolation to causal dynamics modeling, providing a prerequisite for unified world simulators.Source: §2.1 · p. 5

2.2 Modal Consistency

Modal consistency is framed as aligning heterogeneous modalities in a physically complete, logically self-consistent representation. The section traces how geometric mismatch motivates changes in generative mechanisms, multimodal architectures, feedback alignment, and inference-time reasoning.

2.2.1 Theoretical Foundations

  • The Platonic Representation Hypothesis treats images and text as projections of a shared physical latent state, but entropy asymmetry and manifold nonisomorphism make perfect isometric alignment ill-posed.The section contrasts efficient one-pass amortized inference with latent-space search and planning, which use additional inference compute to correct physical drift and address unseen counterfactuals.Source: §2.2.1 · pp. 7–8

2.2.2 Discrete Sequences vs. Continuous Manifolds

  • Discrete AR unifies generation through tokens but can amplify quantization error exponentially, whereas continuous Flow Matching uses deterministic transport and Rectified Flow yields approximately linear error growth.AR's codebook utilization decays as dimension increases, and recursive error can grow as ∥δT∥≈L^T∥ϵ0∥ [54]; FM's rectified path has L≈1 and ∥δT∥≈T · ϵstep, supporting high-fidelity generation in few steps.Source: §2.2.2 · pp. 9–10

2.2.3 Architectural Evolution

  • Dual-tower models such as CLIP and ALIGN align modalities for retrieval but lack fine-grained interaction; connector methods such as Flamingo and BLIP/BLIP-2 bridge visual encoders to LLMs.Early fusion risks optimization instability, while LLaVA's projection can discard image texture information needed for generation.Source: §2.2.3 · pp. 10–11
  • MM-DiT decouples text and image weights while sharing attention, reducing gradient conflict from over 50% in AR paradigms to approximately 30%.In Stable Diffusion 3.5 Large, this design is associated with stronger instruction following and physical fidelity than LLaVA on complex typography rendering and long-text comprehension tasks.Source: §2.2.3 · pp. 11–12, Figure 8 · p. 12

2.2.4 Intent Alignment via RL

  • Preference alignment adds stepwise denoising supervision through SPO and VisualPRM, physics-aware feedback through PhyGDPO, and VLM-based generation-refinement loops such as SRUM.For AR generation, reward-guided token selection with KL regularization suppresses repetition and garbled text in long sequences [42, 62].Source: §2.2.4 · pp. 12–13

2.2.5 Cognitive Loop via Test-time Compute

  • Because amortized pattern matching can hallucinate on counterfactual tasks requiring multi-step deduction, test-time search combines MCTS with verifiers to check and correct generated trajectories.UniGen and EvoSearch use multi-step reasoning, while VisualPRM verifies logical nodes in denoising trajectories.Source: §2.2.5 · p. 13

2.3 Spatial Consistency

This section examines spatial consistency as the geometric basis for grounding semantic representations in physically constrained 3D structure across views, supporting navigation and interaction. It traces approaches from 2D proxies through implicit fields, explicit primitives, and hybrid geometric-diffusion methods, including extensions toward physical interaction and action-consistent world models, geometric attention, and large-scale data strategies.

2.3 Spatial Consistency

  • Spatial consistency grounds semantic latents in a physically constrained 3D manifold, recovering geometry from sparse 2D observations while avoiding structural artifacts such as the Janus problem.Figure 9 illustrates geometric coherence across front, side, and top-down views.Source: §2.3 · p. 14, Figure 9 · p. 14

2.3.1 Geometric Decomposition of Consistency

  • Local topological consistency constrains neighboring surface attributes by a Lipschitz condition, implemented through regularization such as IGR’s Eikonal constraint and RegNeRF’s smoothness loss.These methods aim to produce smooth, physically reasonable surfaces, including under sparse views.Source: §2.3.1 · pp. 14–15
  • Global geometric consistency requires epipolar equivariance across viewpoints; SyncDreamer’s 3D cost volume and MVDream’s multi-view attention address incompatible geometry and the Janus problem.These mechanisms enforce alignment or internalize geometric constraints to preserve global topological uniqueness.Source: §2.3.1 · p. 15

2.3.2 Theoretical Formulation

  • The framework treats spatial consistency as coupled inverse differential problems governed by rendering, generation, and motion laws: RTE, SDE, and Lagrangian transport.NeRF uses volume integration for continuity, while 3DGS uses Gaussian primitives for real-time rendering; reverse SDE learning recovers geometric structures from noise, and Lagrangian transport conserves features along motion.Source: §2.3.2 · pp. 15–16

2.3.3 2D Proxy Manifold & Domain Mismatch

  • 2D proxy models lack 3D rotation equivariance, making large viewpoint changes prone to nonphysical distortion; occlusion also breaks spatial continuity and prevents strict 3D consistency.Convolution-based methods model translation equivariance but not SO(3), while occlusion-induced depth changes make optical flow non-differentiable and PDE constraints fail.Source: §2.3.3 · pp. 16–18

2.3.4 Implicit Continuous Fields

  • Implicit fields provide continuous geometry, while Zip-NeRF combines multisampling and feature smoothing to reduce moiré and high-frequency artifacts; volume sampling costs O(Nsamples), and continuous fields struggle with topological fractures.Mip-NeRF’s Gaussian-volume features act as an adaptive low-pass filter, and Eikonal-regularized SDFs support smooth closed surfaces.Source: §2.3.4 · pp. 18–19

2.3.5 Explicit Lagrangian Primitives

  • Explicit 4D primitives evolve through Lagrangian particle tracking, Eulerian tensor decomposition, or canonical deformation; 4D-GS reduces storage from O(N4) to O(N2), but discrete primitives struggle with fractures and fusions.PhysGaussian embeds continuum-mechanics constraints, while canonical deformation predicts offsets from a static base.Source: §2.3.5 · pp. 19–20

2.3.6 Generative Statistical Priors

  • Under sparse open-world observations, video diffusion priors support 3D consistency through geometric constraints; SDS derives gradients from pretrained diffusion scores, while VSD addresses SDS over-smoothing and recovers texture detail.VSD introduces a variational distribution and minimizes its KL divergence from the prior distribution.Source: §2.3.6 · pp. 20–21
  • MVDream replaces spatial self-attention with 3D correspondence attention using camera parameters (R, T) to align features across generated views and achieve soft geometric consistency.The passage motivates the approach by noting that pure 2D priors are difficult to guarantee multi-head consistency.Source: §2.3.6 · p. 21
  • A synthetic-real-generative data strategy aggregates tens of millions of internet-sourced 3D assets in Objaverse-XL and uses G-Objaverse RGB-D-Normal triplets to train LGM.G-Objaverse's triplets are produced through a physical rendering pipeline and are described as a standard training source for Large General Reconstruction Models.Source: §2.3.6 · p. 21
  • MVImgNet and Co3D-v2 provide real-world object-centric video sequences that reduce synthetic-to-real appearance and texture discrepancies despite largely lacking dense geometric ground truth.The datasets contain millions of captured sequences.Source: §2.3.6 · p. 21
  • See3D combines video diffusion models such as SV3D with Dust3R geometric inference and LGM reconstruction to create a closed-loop engine for expanding pseudo-3D assets.The pipeline synthesizes large-scale pseudo-3D videos before geometric inference and rapid reconstruction.Source: §2.3.6 · p. 21
  • Spatial modeling progressed from 2D proxy fitting through 3D implicit representations to efficient, scalable 3D Gaussian Splatting, with current hybrid architectures combining explicit primitives and implicit diffusion priors.The hybrid approach draws on the complementary strengths of the two representation types.Source: §2.3.6 · p. 21
  • Future spatial consistency work aims to connect symbolic logic with geometry, model physical interaction and action-consistent scenes, and move toward end-to-end native 4D streaming.The proposed direction includes differentiable mappings from LLM symbolic logic to numerical parameters, RL in the generative loop, and real-time inference with compressed 4D tokens.Source: §2.3.6 · p. 21

2.4 Temporal Consistency

Temporal consistency turns a geometrically complete static world into a model of future trajectories, governed by physical constraints and causal logic. The section traces how video-generation approaches address temporal stability, physical compliance, and long-range causal reasoning.

2.4.1 From Frequency Stability to Physical Compliance

  • FVD mainly measures spatial feature-distribution similarity and can miss temporal flicker, whereas VCD compares video features in the temporal frequency spectrum.VCD uses a feature extractor and a time-axis STFT; real motion continuity and high-frequency energy fluctuations expose temporal inconsistency.Source: §2.4.1 · p. 23, Figure 15 · p. 23
  • VCD < 0.3 is reported for frequency-reward fine-tuning, while Veo 3 exceeds 70% success on zero-shot physical-interaction tasks such as domino toppling.The section presents these as steps from reducing temporal jitter toward evaluating physical-law compliance and causal reasoning.Source: §2.4.1 · pp. 23–24, Table 3 · p. 24

2.4.2 Latent Temporal Inflation

  • Temporal inflation freezes pretrained 2D spatial layers and inserts temporal attention, but its first-order Markov relaxation can cause semantic drift beyond 16 frames and smearing when motion exceeds 20% of screen width.Zero-shot attention injection reuses the first frame’s Key/Value features, while depth or pose can provide spatial anchors; frozen 2D topology also struggles with 180-degree viewpoint changes or new content.Source: §2.4.2 · pp. 24–25
  • Temporal inflation models can lose high-frequency energy above 15Hz; GFN and AFNO use Fourier-domain mixing, while FastInit predicts initial noise from preceding-frame features.Despite these corrections, frozen spatial convolutions leave inflation models prone to severe texture stretching under large viewpoint changes or newly appearing content.Source: §2.4.2 · pp. 24–25

2.4.3 Discrete Autoregressive Modeling

  • Causal 3D compression in MagViT-v2 prevents future-frame leakage, while autoregressive generation faces accumulating errors and discretization limits that motivate continuous and hybrid approaches.VAR uses next-scale prediction, and FramePack combines context packing, anti-drift sampling, and PFP; high-dimensional STE training can cause codebook collapse, while Show-o jointly models discrete and continuous tokens.Source: §2.4.3 · pp. 25–26

2.4.4 Unified Spatiotemporal Modeling via DiT

  • Native 3D DiT models video as T × H × W patches and uses full-sequence attention to capture non-local physical interactions; Flow Matching supports smooth recovery of details from noise.The section contrasts this continuous-latent approach with temporal inflation’s fragmentation and discrete autoregression’s quantization loss.Source: §2.4.4 · p. 26
  • Video-TTT targets linear O(N) long-video memory, Pyramid Flow reduces high-quality generation cost by 5–10 times, and TeaCache reports 2–3 times acceleration with zero image-quality loss.TeaCache uses the high similarity of adjacent diffusion-step features, reported as a Pearson correlation coefficient greater than 0.98.Source: §2.4.4 · pp. 26–27

2.4.5 Logical Consistency and Causal Reasoning

  • Image-assisted reasoning in Mini-O3, VisCoT, and RE-CAP supports logical inference, while Chain-of-Frame methods plan causal key states before synthesizing intermediate frames.Think Sound adds audio cues to constrain video evolution and align causal structures across modalities, mitigating logical degradation in long videos.Source: §2.4.5 · p. 28

2.5 Outlook of the Consistencies

The three consistencies have developed distinct computational engines, but optimizing them independently cannot produce a coherent world simulator without a shared cognitive substrate. The outlook is architectural unification, where semantic understanding, geometric structure, and causal reasoning co-emerge within one parameter space.

2.5 Outlook of the Consistencies

  • Modal Consistency advanced semantic translation, Spatial Consistency moved from coarse 2D approximations to explicit 3D primitives, and Temporal Consistency progressed toward causal world simulation.Temporal Consistency evolved from simple frame interpolation.Source: §2.5 · p. 28
  • Independent optimization of specialized capabilities cannot yield a coherent world simulator without a shared cognitive substrate.The challenge shifts from refining isolated components to achieving architectural unification.Source: §2.5 · p. 28
  • Future world models must co-emerge semantic understanding, geometric structure, and causal reasoning within a single parameter space.This requirement motivates the emergence of UMMs.Source: §2.5 · p. 28

3 Initial Integration of Multiple Consistencies · 3.1 The Rise of Large Multimodal Models

The rise of Large Multimodal Models marks a shift from single-task specialists toward unified cognitive systems that integrate modalities, space, and time. The section describes the LLM as their reasoning core and traces how planning and tool use extend their capabilities.

3.1 The Rise of Large Multimodal Models

  • LMMs shift from single-task specialists toward general cognitive entities that integrate Modal, Spatial, and Temporal Consistency in a unified system.The paper presents LLaVA and GPT-4V as examples of this paradigm shift.Source: §3.1 · p. 28

3.1.1 LLM as a Core Cognitive Base

  • Modern LMMs use a pre-trained LLM as a reasoning engine, mapping heterogeneous modalities into its word-embedding space through translators for semantic alignment.Examples of translators include visual connectors and adapters.Source: §3.1.1 · pp. 28–29
  • Visual encoders and projection layers convert image patches into text-aligned visual tokens, which are concatenated with text embeddings as a hybrid input sequence.LLaVA and its subsequent improvements use an MLP or linear projection layer for this bridging step.Source: §3.1.1 · p. 29
  • Aligning visual tokens with text tokens lets the LLM’s self-attention mechanism compute associations between them as it does between text tokens.The paper describes this as the physical significance of alignment.Source: §3.1.1 · p. 29
  • BLIP-2 and Flamingo use learned-query bottlenecks to compress spatial patches into abstract semantic tokens, reducing computational overhead and limiting retained visual information.The paper relates this semantic pooling to the Information Bottleneck hypothesis; DeepSeek-VL and InternVL experiments indicate pre-alignment can form a cross-modal physical manifold that supports logical consistency in unseen scenarios.Source: §3.1.1 · p. 29

3.1.2 Cognitive Evolution as a Multimodal

  • LMMs treat the LLM as a multimodal operating-system kernel that schedules resources, coordinates logic, and invokes specialized modules on demand.This extends the LLM beyond feature processing and is contrasted with traditional end-to-end mapping.Source: §3.1.2 · p. 29
  • VisProg and ViperGPT turn visual queries into executable programs, while PaLM-E and Voyager use perception feedback to search latent action spaces.Recursive task decomposition and feedback support long-term consistency in dynamic environments.Source: §3.1.2 · p. 29
  • Visual ChatGPT and HuggingGPT use ReAct to call expert tools during generation, while Chameleon and Auto-GPT iteratively refine outputs using feedback evaluation.Tools can verify spatial relations or redraw textures; feedback can measure mutual information or geometric constraint deviation ∆ϕ.Source: §3.1.2 · p. 30, Figure 17 · p. 30

3.2 Integration of Modal and Spatial Consistency

This section presents modal-spatial integration as a bridge toward physical world simulators, combining language-controlled spatial relations with geometric grounding through complementary pixel-, view-, volume-, and reinforcement-learning approaches. It also describes RL-based alignment strategies using region-specific preference feedback, spatial rewards, inference-time optimization, and reasoning.

3.2.1 Pixel Space Manipulation

  • Pixel-space editing combines semantic updates with structural constraints to change image content while preserving spatial layout.Gradient decoupling protects the original layout, while attention injection gates edits using source-image spatial maps. Joint image-text modeling captures pixel-level spatial constraints, and video supervision can support object permanence and learned physical dynamics.Source: §3.2.1 · pp. 32–33

3.2.2 View Space Mapping

  • View-space mapping conditions generation on camera pose and depth, combining pretrained image-generation capability with geometric controls for multi-view consistency.A frozen backbone with a trainable bypass preserves semantic generation, while epipolar attention constrains cross-view interactions and parallel RGB-normal supervision with camera-projection-based noise supports geometrically controllable generation.Source: §3.2.2 · pp. 34–35

3.2.3 Volume Space Representation

  • Volume-space methods make spatial consistency intrinsic to 3D representations and use video-model temporal structure as a dynamic prior for multi-view geometry.SV3D uses temporal attention as a soft epipolar constraint and distills the resulting geometrically continuous views into static 3D assets, addressing viewpoint conflicts.Source: §3.2.3 · p. 35
  • Volume-space alignment links 3D geometry with semantic representations through both contrastive alignment and generative interaction.Contrastive methods align 3D features with text and image spaces, while 3D tokenization lets language models read and generate geometry; semantic fields support language queries and topology-preserving edits.Source: §3.2.3 · pp. 36–37

3.2.4 Reinforcement Learning for Modal-Spatial Alignment

  • Modal-spatial reinforcement learning addresses compositional instruction failures by progressing from object-level rewards toward differentiable refinement and local image-text optimization.DDPO uses detector-based IoU rewards, whose sparsity limits complex attribute binding; AlignProp back-propagates spatial errors to the denoising network, while R-DPO decomposes images and prompts into local pairs.Source: §3.2.4 · pp. 37–38
  • The described region-level preference setup restricts modal attributes to matching spatial regions and defines local preferences using winning and losing image crops with a sigmoid.The local preference dataset is denoted Bk; the crops are xw and xl.Source: §3.2.4 · p. 38
  • SPO separates spatial-IoU and semantic optimization across early and late denoising, while DRaFT maps VLM critiques into dense pixel rewards for subtle spatial-modal errors.This shifts RL alignment from discrete boxes toward continuous pixel fields.Source: §3.2.4 · p. 38
  • TTPO iteratively updates latent z at inference using a lightweight reward model rather than heavy retraining, offering a compute-for-quality path for counter-intuitive generative tasks.Its effectiveness is primarily validated in image restoration tasks.Source: §3.2.4 · p. 38
  • Inference-time RL is presented as fusing modality and space, while Layout-CoT plans, aligns, and verifies layouts before decoding only chains that pass verification into pixels.This moves System-2 logic verification to the front end to address interpenetration and spatial misalignment.Source: §3.2.4 · p. 38

3.3 Integration of Modal and Temporal Consistency

This section examines methods for integrating modal semantics with temporal dynamics, treating video generation as a probabilistic simulation of spatiotemporal causality in which instructions guide semantic content and object permanence and physical laws support coherent evolution. It covers structured attention, unified perception-generation architectures, reward-based alignment, and reinforcement learning approaches using physical verification and exploration to address limitations of imitation in embodied action.

3.3 Integration of Modal and Temporal Consistency

  • Modal and temporal consistency jointly connect instruction-following semantics with physically coherent frame evolution, addressing flickering, temporal logic chaos, and long-video collapse.The section describes semantic consistency as preventing cross-modal drift and dynamics consistency as supporting object permanence and physical conservation.Source: §3.3 · p. 38

3.3.1 End-to-End Scalable Modeling

  • End-to-end video models scale model and data together to fit multimodal-to-video distributions, with Flow Matching using ODE trajectories to improve sampling efficiency and temporal coherence.Rectified Flow reduces transport curvature, enabling physical dynamic textures in few steps and addressing structural collapse during long-term DDPM sampling.Source: §3.3.1 · pp. 39–40
  • Causal 3D encoding, native 3D attention, and coarse-to-fine cascades support unidirectional temporal logic, long-range spatiotemporal dependencies, and long-video generation.Causal 3D VAEs prevent future-information leakage; native 3D attention captures joint dependencies, while cascades mitigate logic drift and VRAM pressure.Source: §3.3.1 · pp. 40–41
  • Autoregressive video generation treats video as a causal sequence of discrete latent tokens, while hybrid approaches combine AR planning with diffusion refinement to address long-video error accumulation.NoVA predicts continuous features with a diffusion decoder, and RFLAV applies rolling flow matching to refine coarse AR tokens for physically consistent long-duration generation.Source: §3.3.1 · pp. 41–42

3.3.2 Explicit Structured Control

  • Explicit structured control projects video dynamics onto interpretable conditions such as depth, optical flow, or skeletons to improve controllability beyond text-only interaction.Motion vectors, trajectories, and storyboards provide physical or narrative constraints; VAST [266] uses layout-and-pose constraints to suppress identity drift in long sequences.Source: §3.3.2 · pp. 43–44
  • Start-end frame anchoring reframes video generation as interpolation, specifically solving a Brownian Bridge with start and end images as boundary conditions; dynamic instruction injection enables multimodal interactive control within constrained spaces.KeyVID extracts audio peaks with ImageBind as implicit keyframes for automated audio-visual synchronization.Source: §3.3.2 · p. 44
  • Orthogonally decoupling appearance and motion features helps preserve identity during action changes and reduce feature entanglement.Animate Anyone and MagicAnimate separate reference appearance from motion, while Moonshot and CCEdit use decoupled attention branches to limit interference.Source: §3.3.2 · pp. 44–45
  • Swap Attention uses role swapping within 3D windows so spatial information guides time and temporal feedback supports texture and optical-flow coherence, reducing motion freezing.Source: §3.3.2 · p. 45

3.3.3 Unified Comprehension and Generation Symbiosis Architecture

  • Unified architectures treat perception and generation as a joint model, using shared representations and token-based methods for Any-to-Any transformation.Gaia-1 and Phenaki use unified discrete video, control, and text tokens; Unified Discrete Diffusion and Show-O support bidirectional denoising and shared understanding-generation weights.Source: §3.3.3 · pp. 46–47
  • Shared representations also bridge perception and simulation through HERMES’s BEV hub and frozen-MLLM adapters with bidirectional LLM-video projections.MERV dynamically combines frozen encoders such as CLIP, DINOv2, and VideoMAE to balance semantic, structural, and action features.Source: §3.3.3 · pp. 46–47

3.3.4 Reinforcement Learning for Modal-Temporal Alignment

  • Reinforcement learning jointly optimizes modal alignment and temporal evolution, addressing SFT’s tendency to trade semantic fidelity against dynamic coherence.The proposed multi-dimensional reward manifold guides optimization toward the semantics-temporal Pareto frontier.Source: §3.3.4 · p. 47
  • Preference-based methods encode semantic-temporal dependencies in rankings, while VideoDPO penalizes motion collapse and T2V-Turbo distills aesthetic and temporal rewards.VideoDPO uses KL constraints to favor high-dynamic, high-fidelity outputs; T2V-Turbo combines HPSv2 and InternVideo2 signals.Source: §3.3.4 · p. 48
  • Self-refinement aligns instructions and temporal evolution through generation, evaluation, and correction, including hierarchical frame-level rewards and Video-STaR’s rejection-sampled fine-tuning.Hierarchical rewards score first-frame image quality and subsequent-frame motion coherence; Video-STaR filters samples for both instruction accuracy and natural action.Source: §3.3.4 · p. 48
  • Universal reward models combine semantic and temporal evaluation: VPO optimizes both, while VideoScore evaluates causal event order beyond noun alignment.VPO combines Video-LLM semantic alignment with optical-flow smoothness; VideoScore can distinguish whether an instructed event occurs in the correct causal sequence.Source: §3.3.4 · p. 48
  • VLA supervised fine-tuning fits demonstration averages and can cascade under out-of-distribution shifts, while RL uses digital-twin feedback and simulation exploration to pursue feasible actions in unseen states.TwinRL-VLA and RL-VLA3 optimize long-term reward; TwinRL verifies action sequences in a physics engine and expands exploration beyond demonstrated states.Source: §3.3.4 · p. 49

3.4 Integration of Spatial and Temporal Consistency

This section examines how spatial–temporal consistency supports dynamic object permanence and physical 4D generation, covering approaches from implicit prior fusion and geometric anchoring to unified video-derived spatiotemporal representations. It also describes trajectory-based reconstruction, end-to-end geometric recovery, and reinforcement-learning methods using decoupled rewards and explicit physical costs to align spatial fidelity with temporal coherence.

3.4 Integration of Spatial and Temporal Consistency

  • Spatial–temporal fusion requires objects to retain rigid form, physically consistent motion, and intrinsic properties through occlusion and viewpoint change, enabling dynamic object permanence.Figure 28 illustrates an occluded subject re-emerging with unchanged attributes.Source: §3.4 · p. 49, Figure 28 · p. 50

3.4.1 Implicit Spatiotemporal Learning

  • Implicit spatiotemporal learning distills video priors into 4D fields without full-parameter 3D-data fine-tuning, combining geometric and dynamics priors while addressing gradient conflicts and distribution mismatch.Solutions include trajectory mapping, SDE variance reduction, time-varying frequency modulation, and feature-level manifold alignment.Source: §3.4.1 · pp. 50–52
  • Approximately 40% less stochastic jittering was reported for NVS-Solver, which corrects variance inflation from heterogeneous gradients with high-order SDE approximations and variance-reducing sampling.The passage also reports improved sharpness and spatiotemporal consistency.Source: §3.4.1 · p. 51

3.4.2 Explicit Geometric Anchoring

  • Explicit geometric anchoring fixes space with 3D proxies and parameterizes time as continuous SE(3) poses, deriving consistency from geometry rather than network memory.Methods use point-cloud rendering and camera trajectories, geometric tokens and cross-frame associations, or trajectory constraints such as acceleration regularization to guide physically plausible motion.Source: §3.4.2 · pp. 52–54

3.4.3 Unified Spatiotemporal Representation

  • Unified representations reconstruct 4D fields with factorized planes, trajectory-based rendering, and deformable Gaussian primitives, encoding spatial structure and temporal evolution together.Six-plane factorization reduces representation complexity from O(N4) to O(N2); canonical-space deformation fields model changing Gaussian positions and rotations.Source: §3.4.3 · pp. 54–55
  • Trajectory-centric foundation models use dense trajectory fields as an intermediate representation linking 2D video flow to 4D physical space for generalizable inference.Figure 31 depicts trajectory lifting into a 4D canonical space for volumetric reconstruction.Source: §3.4.3 · p. 56, Figure 31 · p. 56
  • Trace Anything and SpatialTracker lift 2D tracks into 3D trajectories, while OmniMotion maps video pixels to canonical 3D space for occlusion-aware lifecycle tracking.These approaches can turn monocular video into 4D pseudo-ground truth; OmniMotion’s continuous mapping supports plausible trajectories through visibility breaks.Source: §3.4.3 · p. 56
  • MASt3R jointly predicts 3D point clouds and camera motion from image pairs, while VGGSfM uses dense trajectories to recover globally consistent structure in long videos.MASt3R does not require camera parameters; VGGSfM addresses cumulative errors and dynamic-object interference through differentiable global structure-from-motion.Source: §3.4.3 · p. 57

3.4.4 Reinforcement Learning for Spatial-Temporal Alignment

  • Reinforcement learning addresses SFT’s late-stage spatial collapse and temporal causality breaks by optimizing composite rewards for image fidelity and motion coherence.The section groups approaches into global mixed-reward feedback and explicit physical costs.Source: §3.4.4 · p. 57
  • T2V-Turbo-v2 distills separate aesthetic and motion-coherence rewards to mitigate motion averaging, while DR-Tune dynamically adjusts spatiotemporal gradient weights during denoising.T2V-Turbo-v2 uses HPSv2 for single-frame aesthetics and InternVideo2 for dynamic coherence; DR-Tune aims to improve temporal consistency without sacrificing frame fidelity.Source: §3.4.4 · p. 57
  • VistaDPO separates semantic, temporal, and spatial alignment, while InstructVideo uses flickering and optical-flow consistency costs to reduce long-video jitter.The hierarchical design targets temporal time-skipping without damaging spatial textures; InstructVideo’s explicit costs provide gradient signals for smooth pixel flow.Source: §3.4.4 · p. 57

3.5 Preliminary Emergence of World Models

The section describes World Models as beginning to emerge through synergistic modal, spatial, and temporal consistency: connecting multimodal interfaces, geometric structure, and causal evolution in an internal simulator. It traces this development from video-generation architectures to interactive and embodied systems that model physical constraints.

3.5 Preliminary Emergence of World Models

  • Modal consistency supplies interaction interfaces, spatial consistency builds geometric structure, and temporal consistency drives causal evolution in the emerging internal simulator.The section characterizes this as a shift beyond single-dimension modeling and pixel generation, and says benchmarks have quantitatively verified the synergy.Source: §3.5 · p. 58

3.5.1 From Benchmark Establishment to Diverse Evolution

  • Sora uses large-scale spacetime-patch training to exhibit perspective constancy and causally plausible interactions without explicit 3D constraints.The section links its emergence to scaling laws and describes interoperability between spacetime and modality; it says these deductive traits suggest a world simulator.Source: §3.5.1 · pp. 58–59
  • Open-Sora combines alternating spatial and temporal attention with cascade training and video VAE components to support efficient, coherent minute-level generation.The section presents its Spatial-Temporal Diffusion Transformer as balancing within-frame spatial fidelity and across-frame temporal coherence, and argues that architecture matters alongside compute.Source: §3.5.1 · p. 59
  • Interactive models including Genie 1/2/3, LingBot-World, and GameNGen shift from passive video prediction to action-conditioned state transitions.Genie-3 uses latent action tokens; LingBot-World couples semantic instructions with physical attributes and uses spatiotemporal regularization for spatial fidelity at 60 FPS.Source: §3.5.1 · p. 59
  • CogVideoX strengthens inter-frame dependency through 3D RoPE, whereas Wan2.1’s Causal 3D VAE enforces unidirectional temporal flow within latent space, enhancing physical plausibility of dynamic evolution.Gen-3 maintains cinematic-grade lighting and simulates complex physical interactions; HailuoAI uses engine optimizations to resolve structural collapse in complex dynamics. LTX-Video explores multimodal embedding spaces and attention mechanisms to strengthen semantic alignment between textual instructions and visual content.Source: §3.5.1 · p. 59

3.5.2 Combat Loop of Three Consistencies

  • Embodied systems extend the trinity to closed-loop action, while physics-engine Digital Twins constrain trajectories and support sim-to-real transfer.WorldVLA maps language to joint controls, while LingBot-World and 3D-VLA couple semantic instructions, feedback, and geometry; physics engines impose state transitions s_(t+1) = f_physics(s_t, a_t), and SimplerEnv and ManiSkill2 demonstrate transfer with minimal adaptation. The section concludes that trinity synergy supports world simulators that internalize physical laws and enable counterfactual reasoning.Source: §3.5.2 · pp. 60–61

4 Challenges, Benchmarks, and Outlook

The section argues that general world models need physically grounded, robust, interactive simulation, while current evaluation misses important causal and long-range failures. It surveys benchmarks for modality, spatial, and temporal consistency, introduces CoW-Bench’s task-driven evaluation protocol, and envisions world models as general simulators.

4.1 Core Challenges from Preliminary Fusion to True Unification

  • True unification requires physically grounded dynamics, robust long-range causal chains, interactive control, and models that encompass both physical and digital worlds.Current likelihood objectives can yield visually plausible but physically incorrect motion; hierarchical dynamics are proposed to mitigate long-horizon failure. Users should be able to intervene through forces and material changes, while multi-agent and GUI settings require social causality and cross-domain simulation.Source: §4.1 · pp. 61–62

4.2.4 Limitations of Existing Benchmarks & Design Rationale of Our Benchmark

  • Because perceptual metrics such as FID and FVD miss logical fractures, CoW-Bench evaluates three consistencies and their pairwise fusions across six task categories and 18 subtasks.Each subtask has five human checklists designed to provide fine-grained criteria for interpreting complementary failure modes.Source: §4.2 · p. 62, §4.2.4 · p. 65

4.2.1 Modal Consistency: From Symbol Mapping to Knowledge Synergy

  • Modal benchmarks have moved beyond shallow CLIP co-occurrence toward knowledge internalization, bidirectional text–image reasoning, and testing whether correct understanding transfers to generation.WISE uses counterfactual samples and WiScore, ROVER tests reciprocal Text ↔ Pixel reasoning, and UniSandbox exposes an understanding–generation gap that explicit CoT can help bridge.Source: §4.2.1 · pp. 62–63

4.2.2 Spatial Consistency: From Visual Similarity to Topological & Physical Verification

  • Spatial evaluation now tests scene-structure reasoning and physical validity rather than relying only on visual similarity.VR-Bench finds weaker pure-visual spatial reasoning than text-assisted reasoning, while VBench measures object constancy and scene-layout graph edits; PhysBench and PhysDreamer use physics engines to detect penetration and floating artifacts.Source: §4.2.2 · p. 63

4.2.3 Temporal Consistency: From Inter-frame Smoothness to Logical Causal Evolution

  • Temporal benchmarks examine time-varying knowledge, physical state trajectories, executable reasoning sequences, and long-range failures that reveal reliance on text priors.TempViz uses 7.9k prompts; TiViBench tests video-based physical problem solving, and V-ReasonBench uses optical flow to monitor motion mutations. GGBench checks construction order, while MME-COF and WEAVE expose hallucinations and permanence failures; Thinking with Video cautions that reasoning may rely on LLM text priors.Source: §4.2.3 · pp. 63–64

4.2.4 Limitations of Existing Benchmarks & Design Rationale of Our Benchmark

  • Existing benchmarks can reward smooth-looking violations of physics and overstate generalization because models may memorize familiar data distributions.MLLM referees can miss fine-grained physical errors, and current visual evaluation lacks hard simulation-ground-truth verification. Out-of-distribution tests expose failures in long occlusions, multi-object interactions, and counter-intuitive settings.Source: §4.2.4 · p. 64
  • Short-sequence tests and static prediction tasks miss error accumulation and cannot establish whether models support causal intervention.Without process verification or physical correction, small errors can amplify over time; current evaluations also lack parameterized interventions to test counterfactual reasoning.Source: §4.2.4 · pp. 64–65

4.3 Ultimate Outlook: General World Simulator

  • The envisioned general world simulator would instantiate physical laws and narrative rules on demand, serving scientific exploration and embodied-AI training while integrating physics, society, and emotion.The proposed outlook treats the world as a source of constraints and intelligence as a generator of hypotheses that evolve together in differentiable spacetime.Source: §4.3 · p. 65

5 CoW-Bench … 5.3 Comparison with Existing Benchmarks

CoW-Bench turns the Trinity of modal, spatial, and temporal consistency into a controlled, diagnostic evaluation framework for world-model simulation. It contrasts generative simulation with existing perception and question-answering benchmarks.

5.1.1 Dataset Construction

  • The task blueprint covers modal, spatial, and temporal consistency plus their pairwise integrations, while a single-consistency control protocol constrains unrelated variables to isolate failure modes.Confounds such as entity count, background complexity, camera movement, motion magnitude, and occlusion conditions are explicitly constrained.Source: §5.1.1 · p. 66, Table 4 · p. 66
  • Reasoning-capable models construct seed instances as structured, verifiable units linking a text prompt and initial state to expected image and video outputs.The seeds are designed to internalize target constraints and trigger the corresponding failure modes.Source: §5.1.1 · p. 66

5.1.2 Dataset Analysis

  • The audited benchmark contains 1,485 samples across 18 balanced sub-tasks (69–91 each, including 50 hard Maze cases), with human-machine review checking metric bias and semantic alignment.The balance is intended to prevent inflated scores from overfitting to simpler or more frequent task types.Source: §5.1.2 · pp. 66–68
  • CoW-Bench spans instruction lengths of 7.1–74.8 words; 3D-Reconstruct averages 2.1 elements versus 1.6 in single-modal tasks, while Time-State action complexity averages 77.7 words.These measures probe language complexity, visual and cognitive load, and dynamic evolution complexity.Source: §5.1.2 · pp. 67–68

5.2 Evaluation metrics

  • Evaluation treats generated outputs as constraint-satisfaction problems because plausible results can still violate spatial or temporal constraints, unlike failures captured by FID/IS.Examples include rare materials reverting, local edits causing global drift, framewise world reinitialization, occlusion contradictions, and inconsistent multi-view worlds.Source: §5.2 · p. 68
  • CoW-Bench combines 18 metric families with 16 reusable atomic checks, then scores each invoked dimension from 0–2 and evaluates sequences using four chronologically sampled frames in a 2×2 grid.Atomic checks support diagnosis and cross-task comparison; task-family scores aggregate sub-task averages with equal weighting.Source: §5.2 · pp. 68–70

5.3 Comparison with Existing Benchmarks

  • CoW-Bench evaluates active world simulation, contrasting UniBench’s multimodal perception and MANBench’s VQA accuracy with dynamic constraint satisfaction.Its fine-grained checks target physical failures such as objects disappearing after occlusion, rather than relying only on semantic alignment.Source: §5.3 · pp. 71–72
  • GPT-4V exhibits pronounced failures on cross-consistency tasks, particularly modal–temporal coupling (M × T), despite strong cognitive reasoning capabilities.The paper attributes benchmark difficulty to entangled spatiotemporal dynamics and interacting physical constraints, rather than abstract problem-solving alone.Source: §5.3 · p. 72

5.4 Main Results · 5.5 Single-Axis Consistency · 5.6 Cross-Axis Consistency

CoW-Bench results show strong performance on static or single-view settings but sharp drops when models must preserve constraints across time and space. Across its consistency axes, the benchmark distinguishes visual plausibility from reliable semantic binding, structured evolution, and persistent world-state maintenance.

5.4 Main Results

  • GPT-image-1.5 ranks first overall, ahead of Nano Banana Pro and GPT-image-1, while open-source video generators lag on most consistency-sensitive tasks.Models perform well on static or single-view settings, but cross-consistency tasks expose failures to maintain a stable world as changes unfold.Source: §5.4 · pp. 72–73

5.5.1 Modal Consistency Results

  • Modal consistency is limited by unstable identity–attribute binding and quiet relaxation of instructions, even when outputs look plausible.Models may preserve backgrounds but miss the intended edit, satisfy listed constraints without stable referents, or introduce extra entities; SUAT can blend text and image constraints into a confused instruction.Source: §5.5.1 · pp. 73–75

5.5.2 Temporal Consistency Results

  • Temporal continuity is comparatively strong, but rule-guided evolution and ordered stage transitions remain fragile.Models may keep a subject and scene stable while missing process direction, pacing, intermediate states, or stage order; STOR transitions can be discontinuous or reversed, while image models perform better on SLEV world rules than video models.Source: §5.5.2 · pp. 75–76

5.5.3 Spatial Consistency Results

  • Spatial plausibility is stronger than executing directional, occlusion, and cross-view constraints consistently.Models often achieve coherent planar layouts and depth ordering, yet struggle with directional grounding and visible-part evidence; plausible single-view 3D does not guarantee persistent scene structure across viewpoints.Source: §5.5.3 · pp. 76–78

5.6.1 Modal–Space Consistency Results: Semantic-to-Geometry Binding

  • Modal–space consistency is bottlenecked by semantic role binding: models may produce coherent geometry while attaching actions or relations to the wrong entity.Avoiding forbidden configurations is often easier than constructing exact positive relations, and weaker models can fail catastrophically on multi-view anchors.Source: §5.6.1 · pp. 78–79

5.6.2 Modal–Time Consistency Results: Executing a Temporal Program

  • Modal–time consistency is strongest in anchoring but weakest in executing instructed attribute dynamics and triggered events.Models can keep an observable anchor stable yet fail to follow the requested direction or rate; video models also often miss event timing or fail to preserve the post-trigger state, and may add unspecified conditions.Source: §5.6.2 · pp. 79–80

5.6.3 Time-Space Consistency Results: Navigation Exposes the Missing World State

  • Time–space consistency exposes failures in goal-directed trajectories despite plausible motion and stable environments.Maze tasks reveal incorrect starts, goals, or shortcuts; under motion, depth-layer updates remain difficult, and loop navigation can expose viewpoint resets despite plausible framewise geometry.Source: §5.6.3 · pp. 80–81

5.7 Sample Analysis

CoW-Bench’s sample analysis uses frame-by-frame physical-state ground truth across six consistency challenges to distinguish generators from world simulators. Its examples show both successful consistency behavior and remaining limits in spatial reasoning and physical evolution.

5.7 Sample Analysis

  • The sample analysis evaluates six core consistency challenges using frame-by-frame physical-state ground truth to quantify differences between generators and world simulators.The criteria move beyond visual quality to examine dataset construction and model capability boundaries.Source: §5.7 · p. 81

5.7.1 Single Consistency Tasks

  • Single-task examples show modal consistency through attribute transfer without morphology leakage, precise localized edits, and correct assignment of multiple characters’ attributes and roles.Examples include transferring butterfly texture and material to a fish, editing only a clock’s hands, and following long-text constraints.Source: §5.7.1 · p. 81
  • Single-task spatial examples preserve relative positions, render books moving into darkness as a drawer closes, and keep room geometry and lighting stable during a viewpoint shift.The viewpoint shift naturally takes a desk lamp out of view while revealing the bed layout.Source: §5.7.1 · pp. 81–82
  • Single-task temporal examples preserve fan blades during rotation, shorten a candle over a simulated hour, and depict a gravity-driven collapse whose ruins remain still.The collapse avoids non-causal jitter such as recovering after falling.Source: §5.7.1 · p. 82

5.7.2 Compound Consistency Tasks

  • Compound modal-spatial examples correctly ground a blue-roof vehicle’s movement and distinguish apples inside a bowl from pears outside, but book-and-pen occlusion drifts during viewpoint changes.The drift indicates difficulty maintaining micro-spatial semantics under dynamic viewpoints.Source: §5.7.2 · p. 83
  • Compound modal-temporal examples maintain a vehicle’s attributes over long movement, execute a sphere’s ordered color changes without bleeding, and light a phone only after its button is pressed.These tasks test instruction fidelity and resistance to semantic drift over long sequences.Source: §5.7.2 · p. 83
  • Compound spatial-temporal examples reproduce motion parallax and complete a bedroom-city loop without spatial collapse, while maze navigation fails to plan a path to the goal.The maze walls remain structurally stable despite the path-planning failure.Source: §5.7.2 · p. 83
  • CoW-Bench’s takeaway is that video models outperform image models in instruction following but lack physical evolutionary logic, relying on pixel interpolation that can cause non-physical distortions and topological collapse.The authors describe CoW-Bench as a new standard for evaluating world models.Source: §5.7.2 · p. 83

6 Conclusion

The conclusion defines cross-dimensional consistency as the criterion separating world simulation from texture synthesis, and presents CoW-Bench as a way to diagnose consistency failures. It argues that overcoming those failures requires expressive interaction spaces that connect semantic intent to physical dynamics.

6 Conclusion

  • Current systems fail to bind semantic instructions to geometric roles, maintain identity over long horizons, and preserve environmental permanence during navigation.These are ruptures across Modal-Space, Modal-Time, and Time-Space dimensions, respectively.Source: §6 · p. 84
  • CoW-Bench evaluates video-generation models and UMMs through expert-designed multi-frame checklists, exposing constraint backoff behind plausible-looking textures.It treats consistency as constraint satisfaction by comparing temporally sampled grids against fine-grained atomic checklists, distinguishing visual mimicry from physical simulation.Source: §6 · p. 84
  • Constraint backoff is structural, not merely a consequence of insufficient data or scale: opaque or predefined action spaces cannot ground semantic commitments in physical dynamics.The conclusion argues that consistency violations become nearly inevitable under these interaction constraints, requiring a shift in how interaction is formalized.Source: §6 · p. 84
  • Interaction paradigms progress from opaque Vector-as-Action JEPA and predefined Key-as-Action Genie toward Prompt-as-Action models, including PixVerse-R1.Prompt-as-Action UMMs use a semantic compiler to translate natural-language intent into universal spatiotemporal simulations adhering to the Trinity of Consistency.Source: §6 · pp. 84–85, Figure 50 · p. 84
  • A world model that fails cross-dimensional consistency remains a texture synthesizer, not a simulator, regardless of how visually compelling its output is.The Trinity of Consistency marks the boundary between images that resemble the world and models that understand it.Source: §6 · p. 85

Related papers

Loading 2602.23152v1…