PaperlayerView exact revision on arXiv

Source-linked AI summary

Abductive World Modeling via Causal Representation Learning

Ziqi Liu, Songhan Yang, Linfan Zhou, Jiatong Liu, Lijun Peng, Long Wan, Yinqi Bai

arXiv:2609.36985v1cs.LGcs.AI

TL;DR

World models often predict future states without explicitly representing the latent causes of how the world evolves. AWM uses predicted futures as evidence to infer Entity, Dynamic, and Relation states; it reports improvements over the V-JEPA 2 backbone-only baseline across physical prediction, event reasoning, and action understanding.

  • Problem

    World-model representations often predict future states without explicitly exposing the entities, dynamics, and interactions that explain how the world evolves.

    Source: §1 · p. 1

  • Method

    AWM jointly reasons over current observations and predicted futures, using HASP to organize inferred factors into Entity, Dynamic, and Relation states.

    Source: §1 · pp. 1–2

  • Results

    AWM outperforms the V-JEPA 2 backbone-only baseline across physical prediction, event reasoning, and action understanding, with gains of 10.7% in Physion++ AUROC, 16.8% in CLEVRER question accuracy, and 68.0% in EK100 action Top-1 accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HASP's Entity, Dynamic, and Relation states capture complementary object, motion, and interaction information and show selective responses under targeted interventions.

    Source: §1 · p. 2

    The predicted future is generated solely from observed context and never accesses ground-truth future frames.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

The central challenge of world modeling is to learn representations that capture how the world evolves. However, existing world models predominantly represent future states without explicitly capturing the latent causes underlying their evolution, limiting their ability to reason about why and how the world changes. To address this limitation, we propose Abductive World Modeling (AWM), a framework that learns structured causal representations by abductively inferring latent causes from predicted futures. Specifically, we realize AWM through the Hierarchical Abductive State Pyramid (HASP), which organizes the inferred world state into three complementary components - Entity, Dynamic, and Relation - capturing what exists, how it changes, and how entities interact, respectively. By jointly reasoning over the current observation and its predicted future, HASP abductively infers these latent factors and integrates them into a structured state representation for downstream reasoning. To the best of our knowledge, AWM is the first framework to introduce abductive state inference into latent-space world modeling for learning structured representations of world dynamics. Experiments across physical prediction, causal reasoning, and action understanding demonstrate the effectiveness of our approach. Compared with V-JEPA, a state-of-the-art latent-space world model, AWM improves physical prediction AUROC by 10.7%, causal reasoning accuracy by 16.8%, and action Top-1 accuracy by 68.0%.

1. Introduction

World models often predict future states without explicitly representing the entities, changes, and interactions that explain them. AWM uses predicted futures as evidence to infer these factors, and its evaluation reports gains over the V-JEPA 2 backbone-only baseline across physical prediction, event reasoning, and action understanding.

1. Introduction

  • World models can predict likely futures while leaving their predictive cues entangled and without explicit structure for reasoning beyond prediction.World-state representation matters because its structure and quality shape how well models characterize and reason about evolving environments.Source: §1 · p. 1
  • AWM treats predicted futures as evidence and jointly reasons over them with current observations to infer latent factors underlying world evolution.Its guiding process is to predict forward, then abduce backward.Source: §1 · pp. 1–2
  • HASP hierarchically organizes predictive evidence into Entity, Dynamic, and Relation states at object, temporal, and interaction levels.These states represent what exists, how it changes, and how entities interact.Source: §1 · p. 2
  • Physion++ AUROC improves by 10.7%, CLEVRER question accuracy by 16.8%, and EK100 action Top-1 accuracy by 68.0% over the V-JEPA 2 backbone-only baseline.The reported gains span physical prediction, event reasoning, and action understanding.Source: §1 · p. 2
  • AWM is evaluated across physical prediction, event reasoning, and action understanding, with structured abductive states providing consistent benefits across video reasoning tasks.The introduction identifies Physion++, CLEVRER, and EK100 as the respective evaluation settings.Source: §1 · p. 2

2. Related Work

Related work develops predictive world models, object-centered and relational representations, and causal representation learning. The paper positions AWM as combining structured entity, dynamic, and relational factors with abduction from predicted states.

Latent predictive world models

  • Latent predictive world models learn by predicting future states in representation space, but typically use unified latent states without explicit entity, dynamic, and relational structure.Other predictive approaches model future experience through generation or reconstruction.Source: §2 · p. 2

Object-centric and relational representations

  • Object-centric and structured-dynamics methods capture entities, dynamics, or interactions, but typically focus on individual aspects rather than jointly organizing all three.The cited approaches include object-centered visual representations and models of object interactions.Source: §2 · pp. 2–3

Causal representation learning and abduction

  • Causal representation learning and temporal causal methods study latent factors and mechanisms, while abduction infers latent explanations from observations and consequences.The paper states that these directions generally do not infer structured causes of world evolution from predicted states.Source: §2 · p. 3

3. Method

AWM treats a predicted future latent state as evidence for abductive inference rather than only as a prediction target. HASP progressively organizes that evidence into entity, temporal, and pairwise interaction states, with factor-aligned supervision encouraging each state to represent its intended granularity.

3.1. From Future Prediction to Abductive State Inference

  • HASP maps the observed world state and predicted future evolution into Entity, Dynamic, and Relation states through progressive attribution.The predicted future is generated from observed context alone and does not access ground-truth future frames.Source: §3 · p. 3, §3.1 · p. 4

3.2. Hierarchical Abductive State Pyramid

  • The hierarchy assigns identity before change and interaction, since temporal change depends on identified entities and relations depend on represented entities and their evolution.Entity states are grounded in the observed scene; predicted-future evidence first enters Dynamic attribution and is then propagated to Relation inference.Source: §3.1 · p. 4, §3.2 · p. 4, §3.2.2 · p. 5

3.2.1. Entity Attributor: Attributing Patch Evidence to Entities

  • The Entity Attributor converts distributed visual evidence into persistent object-centered states using the current representation.This establishes a stable entity basis before future-dependent reasoning.Source: §3.2.1 · pp. 4–5

3.2.2. Dynamic Attributor: Attributing Predicted Change to Entities

  • The Dynamic Attributor uses entity-conditioned temporal queries over current and predicted-future evidence to attribute change to each entity.Its output is organized at the entity–time level while retaining entity and current-state information.Source: §3.2.2 · p. 5

3.2.3. Relation Attributor: Attributing Dynamics to Interactions

  • The Relation Attributor constructs pairwise temporal states from already-grounded entities and their entity-specific dynamics.This represents which interaction between two entities can account for their attributed temporal evolution.Source: §3.2.3 · p. 6

3.3. Training Objective

  • Factor-aligned losses supervise Entity, Dynamic, and Relation states at object, entity-time, and pairwise-interaction levels, with optional downstream task supervision.The encoder and latent predictor remain frozen while HASP and task-specific readouts are optimized.Source: §3.3 · p. 6
  • Table 1 compares models across datasets using percentage-valued metrics, with EK100 reported as Top-1/Top-5 and boldface marking each metric-column best.Orca and VideoMAE v2 are not listed on CLEVRER because their released models lack the predictive module required by that evaluation pipeline.Source: Table 1 · p. 7

4. Results

AWM performs strongly across physical prediction, event reasoning, and action recognition, while analyses find that HASP states expose complementary factors and predictions respond selectively to relevant evidence.

4.1. Experiment 1. Main Results

  • AWM improves over V-JEPA 2 across all evaluated task spaces, including Physion++ AUROC by 10.7%, CLEVRER question accuracy by 16.8%, and EK100 action Top-1 accuracy by 68.0%.It also improves Physion++ balanced accuracy by 7.5% and accuracy by 6.5%, CLEVRER option accuracy by 8.3%, and multiple EK100 verb, noun, and action metrics.Source: §4.1 · p. 7

4.2. Experiment 2. Analysis of Attributor Interpretability

  • On Physion++, Entity captures object presence and extent, Dynamic captures motion, and Relation captures contact while retaining distance and time-to-contact information.These results indicate complementary information at object, temporal, and pairwise granularities.Source: §4.2 · p. 8
  • Entity predictions align with ground-truth hands and manipulated objects in EK100 and remain aligned with the same objects across consecutive CLEVRER frames.Figure 2 uses green boxes for ground-truth regions and magenta boxes for matched predicted regions.Source: §4.2 · p. 8, Figure 2 · p. 8
  • On CLEVRER contact prediction and TTC estimation, Relation performs close to the full representation, while Entity and Dynamic are less effective alone.This supports the interpretation that Relation captures most pairwise interaction information, while lower-level states encode complementary object and motion information.Source: §4.2 · p. 9, Figure 3 · p. 9

4.3. Experiment 3. Ablation Study on Intervention Consistency

  • Masking the target Entity slot reduces Physion++ OCP accuracy from 0.7175 to 0.6325 and AUROC from 0.7962 to 0.7391, while masking an irrelevant slot retains 0.7088 accuracy and 0.7834 AUROC.The larger degradation under target-object masking indicates stronger dependence on task-relevant Entity evidence.Source: §4.3 · p. 9, Figure 4 · p. 9
  • Removing temporal variation increases Speed MAE from 0.0670 to 0.1120, decreases Contact AUROC from 0.9837 to 0.9485, and reduces OCP AUROC from 0.7907 to 0.5646.The temporal-static intervention averages and repeats features over time while keeping model components and readouts fixed.Source: §4.3 · p. 10, Table 3 · p. 10
  • On CLEVRER, masking the interacting pair reduces Contact AUROC from 0.9633 to 0.9488 and increases TTC MAE from 0.1306 to 0.1376, a larger change than masking a non-contact pair.The results indicate that Relation states carry pair-specific information used for contact and TTC prediction.Source: §4.3 · p. 10, Table 4 · p. 10

5. Conclusion

AWM uses HASP to represent world dynamics through Entity, Dynamic, and Relation factors, with gains over the V-JEPA 2 backbone across physical prediction, causal reasoning, and action understanding.

5. Conclusion

  • AWM improves over the V-JEPA 2 backbone by 10.7% in Physion++ AUROC, 16.8% in CLEVRER question accuracy, and 68.0% in EK100 action Top-1 accuracy.The conclusion presents these results as evidence of the effectiveness of structured causal representations.Source: §5 · p. 10

AI Use Statement

  • Generative AI tools assisted with literature organization and editorial drafting, while authors checked technical claims, equations, experimental numbers, and code references.The authors take responsibility for the final manuscript.Source: AI Use Statement · p. 11

Reproducibility Statement

  • The authors plan to release source code, dataset-isolated training protocols, and analysis scripts, but not model checkpoints.The scripts export reusable state features and reproduce factor probes and intervention tables without modifying training checkpoints.Source: Reproducibility Statement · p. 11

Funding, Compute, and Compliance

The appendix details HASP’s hierarchical construction of Entity, Dynamic, and Relation states and their downstream readout. It also records the work’s funding, compute, authorship, publication review, intellectual-property, and dataset-use information.

Funding, Compute, and Compliance

  • The work was company-funded, used 32 NVIDIA A800 GPUs, and reports author status, publication review, model and code rights, and licensed research-only dataset use.No new human-subject data was collected; the authors’ views do not necessarily reflect the company’s.Source: Funding, Compute, and Compliance · p. 11

A.1. Predictive Backbone Interface

  • A frozen visual encoder represents the observed context, and a frozen latent predictor estimates future positions without using ground-truth future representations to construct its prediction.The appendix defines batch size, observed temporal positions, spatial visual tokens, and backbone feature dimension.Source: §A.1 · p. 13

A.2.1. Entity Attributor

  • The Entity Attributor uses learnable queries to convert the current representation into object-centered states; the predicted future does not enter this stage.Entity states represent what exists in the scene.Source: §A.2.1 · p. 14

A.2.2. Dynamic Attributor

  • The Dynamic Attributor uses entity-conditioned queries to retrieve observed and predicted-future evidence, attributing temporal changes to individual entities.A residual path preserves current-state information that may not appear as temporal change.Source: §A.2.2 · pp. 14–15

A.2.3. Relation Attributor

  • The Relation Attributor builds temporally evolving pairwise interaction states from Entity and Dynamic states, using symmetric pair features that are invariant to pair ordering.A residual path preserves lower-level dynamic and current-state evidence; the Relation Attributor does not independently attend to the predicted future.Source: §A.2.3 · pp. 15–16

A.3. Hierarchical State Construction

  • HASP’s Attributors consume the structured state from the preceding level, progressively organizing the representation rather than acting as independent feature heads.The hierarchy proceeds from entities to dynamics and then relations.Source: §A.3 · p. 16

A.4. Structured State and Downstream Readout

  • The three state levels are complementary: Entity captures object identity and spatial evidence, Dynamic binds predicted change to entities, and Relation represents evolving pairwise interactions.Residual paths preserve information established at lower levels, and a lightweight task-specific readout operates on the structured state.Source: §A.4 · p. 16

A.5. Factor-specific Training Objectives

HASP applies factor-specific supervision and a sequential inference procedure. Additional Physion++ diagnostics report object-level semantics and show that target-slot interventions alter predictions more than irrelevant or random-slot interventions.

A.5. Factor-specific Training Objectives

  • Factor-specific objectives supervise Entity, entity-time Dynamic, and entity-pair-time Relation states; task-level supervision is added when a downstream task is jointly optimized.Invalid object, time, or pair entries are excluded from the corresponding Physion++ probe losses; the encoder and predictor remain frozen during HASP optimization.Source: §A.5 · pp. 16–17

A.6. Complete Inference Procedure

  • Inference predicts a future latent state, constructs Entity states from current features, then derives Dynamic states and pairwise Relation states before downstream readout.The Dynamic stage uses current and predicted-future representations; Relation states are formed from Entity and Dynamic states.Source: §A.6 · p. 17

B.1. Object Identity Diagnostics

  • Entity slots achieve 99.77% presence F1, 95.79% target-object accuracy, and 92.86% object-type accuracy, with 70.29% identity consistency and a 4.05% ID switch rate.Low center, geometry, and velocity errors further indicate retention of spatial and motion information associated with objects.Source: §B.1 · p. 17, Table 5 · p. 17

B.2. Additional Entity Intervention Diagnostics

  • Target masking changes the OCP prediction boundary: true positives rise from 277 to 325, while true negatives fall from 297 to 181 and false positives rise from 124 to 240.Irrelevant-slot masking and random-slot intervention produce much smaller changes in the confusion pattern.Source: Table 6 · p. 18, §B.2 · p. 18
  • 0.67680 mean absolute logit change under target masking is approximately 3.9 times irrelevant masking (0.17355) and 7.6 times random-slot intervention (0.08895).The analysis concerns OCP predictions and supports selective sensitivity to the target-object Entity representation.Source: Table 7 · p. 18, §B.2 · p. 18

B.3. CLEVRER Relevant-Object Intervention

On CLEVRER predictive questions, removing the query-relevant object degrades accuracy more than removing an irrelevant object. The section also describes the benchmarks used for physical prediction, causal video reasoning, and egocentric action understanding.

B.3. CLEVRER Relevant-Object Intervention

  • 50.18 to 43.46 question accuracy and 71.68 to 66.80 option accuracy follow relevant-object masking on CLEVRER, versus 47.34 and 69.55 when an irrelevant object is masked.The evaluation covers 3,557 predictive questions and 7,114 answer options; the readout and model are held fixed.Source: §B.3 · p. 18, Table 8 · p. 19

C. Details of Assets Used in This Paper

  • Experiments use benchmark-specific settings and shared data splits across compared methods; CLEVRER evaluation is restricted to predictive test questions about future events.Physion++ evaluates physical prediction, CLEVRER causal reasoning, and EK100 verb, noun, and action recognition.Source: C. Details of Assets Used in This Paper · p. 19

C.1. Physical Prediction Dataset

  • Physion++ evaluates physical prediction from videos of interacting objects and varied physical configurations.The experiments use it to assess whether representations capture information relevant to future physical outcomes.Source: §C.1 · p. 19

C.2. Causal Reasoning Dataset

  • CLEVRER is a synthetic video reasoning benchmark covering objects, motion, collisions, and temporal events, including questions about observed and future events.The experiments use it to evaluate Entity, Dynamic, and Relation structure for video-event causal reasoning.Source: §C.2 · p. 19

C.3. Action Understanding Dataset

  • EK100 contains unconstrained first-person recordings of everyday activities, with each action segment labeled by a verb–noun pair defining an action class.The experiments evaluate verb, noun, and action recognition on this benchmark.Source: §C.3 · p. 19

C.4. Data Splits and Evaluation Settings

The evaluation follows each benchmark’s protocol and uses consistent training and evaluation splits across compared methods.

C.4. Data Splits and Evaluation Settings

  • Compared methods use the same training and evaluation splits within each benchmark; CLEVRER evaluation covers only predictive test questions requiring future-event prediction.Physion++ is used for physical prediction, CLEVRER for causal reasoning, and EK100 for verb, noun, and action recognition.Source: §C.4 · p. 19

D. Details of Methods and Experimental Settings

Table 9 summarizes AWM’s main configuration, including its predictive backbone, evaluation configuration, and computational setup.

D. Details of Methods and Experimental Settings

  • Table 9 summarizes AWM’s main configuration, predictive backbone, evaluation configuration, and computational setup.The passage does not provide the table’s configuration values.Source: Table 9 · p. 20

D.1. Details of Experimental Setup

AWM predicts a future latent state and uses HASP to infer Entity, Dynamic, and Relation factors; the setup compares it with matched and external video baselines under the stated evaluation protocols.

D.1. Details of Experimental Setup

  • AWM and V-JEPA 2 share the predictive backbone and input setting, while V-JEPA 2 readouts use representations without explicit Entity, Dynamic, or Relation states.This matched comparison is intended to isolate the effect of HASP under the same predictive backbone.Source: §D.1 · p. 20
  • Orca and VideoMAE v2 lack compatible future predictors, so Physion++ uses encoded ground-truth future frames as substitutes for predicted future latents.On EK100, Orca’s encoder stays frozen with a lightweight attentive readout, whereas VideoMAE v2’s pretrained encoder is fine-tuned with task-specific classification heads.Source: §D.1 · p. 20
  • Orca and VideoMAE v2 have no reported CLEVRER results because the predictive subset requires unobserved future events without corresponding ground-truth future frames.The authors omit these results rather than approximate the missing predicted future representation.Source: §D.1 · p. 20

D.2. Algorithm of Abductive World Modeling

  • HASP infers entities, their changes, and pairwise interactions from current and predicted-future representations to form a structured state for downstream reasoning.AWM uses a frozen encoder and predictor, and keeps them and HASP frozen during evaluation while training lightweight factor probes or task-specific readouts.Source: §D.2 · p. 21, §D.1 · p. 21

Related papers

Loading 2609.36985v1…