PaperlayerView exact revision on arXiv

Source-linked AI summary

AffectWorld: Benchmarking Emotion-Driven Future World Prediction

Changhao Song, Yacong Chen, Hui Gao, Yazhou Zhang, Peng Zhang

arXiv:2610.18305v1cs.CVcs.AI

TL;DR

Existing world-model benchmarks largely leave emotion-driven future evolution untested, despite emotion shaping human actions and social dynamics. AffectWorld evaluates the linked progression from emotion to action to world state, and results across current models show substantial gaps in predicting emotion-driven consequences and responding to emotional interventions.

  • Problem

    Existing world-model benchmarks primarily assess physical or task-oriented futures, leaving whether models predict emotion-driven social evolution largely untested.

    Source: §1 · p. 1

  • Method

    AffectWorld benchmarks world-state, emotion-action, and causal consistency, with an optional counterfactual task that changes emotion while holding scene and event context fixed.

    Source: §1 · p. 2

  • Results

    Across five models, world-state change scores reached only 0.10–0.28 and counterfactual emotion sensitivity scores reached 0.027–0.070, revealing persistent weaknesses in predicting emotion-driven consequences and interventions.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    AffectWorld provides a basis for measuring progress in modeling human behavior alongside physical dynamics.

    Source: §5 · p. 9

    AffectWorld focuses on short everyday videos and may not represent diverse cultural contexts or long-term social dynamics; semantic evaluation may retain judge bias despite human calibration.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The world encompasses both physical environments and human actors. Predicting its evolution requires understanding both physical laws and how emotions shape human actions. However, existing world model benchmarks primarily assess physical plausibility and perceptual quality, leaving emotion-driven world evolution largely untested.We introduce AffectWorld, a benchmark that evaluates the chain Emotion-Action-World State through three tasks: World-State Consistency, Emotion-Action Consistency, and Causal Consistency. It comprises 1,498 everyday video samples annotated with plausible emotions, action tendencies, and atomic world-state constraints. A counterfactual subset of 233 samples tests whether changing an agent's emotion under fixed scene and event conditions yields corresponding changes in actions and consequences. We define eight metrics across the three tasks and an additional counterfactual metric, combining automatic visual assessment with human-calibrated VLM/LLM judgments. Evaluations of five SoTA world models (e.g., Cosmos3) required world-state change scores of 0.100-0.280 and counterfactual emotion sensitivity scores of 0.027-0.070 on a [0, 1] scale. These results reveal substantial gaps in generating the consequences of emotion-driven actions and adapting future predictions to emotional interventions. AffectWorld provides a testbed for advancing world models that capture how emotions shape future physical and social states.

1 Introduction

Existing world-model evaluations largely focus on physical or task-oriented futures, leaving emotion-driven social change largely untested. AffectWorld benchmarks how emotion shapes action and how actions reshape future world states, and evaluations reveal substantial gaps in that chain.

1 Introduction

  • World-model benchmarks assess physical or task-oriented futures, while whether models explain why people act and how actions reshape social futures remains largely untested.Emotion can change action selection, interpersonal distance, and group atmosphere over multiple steps.Source: §1 · pp. 1–2
  • AffectWorld tests three linked tasks—world-state, emotion-action, and causal consistency—and optionally changes only the target agent’s emotion to assess future responses.The benchmark allows multiple plausible emotional futures when their evolution remains causally consistent.Source: §1 · p. 2, Figure 2 · p. 3
  • The benchmark evaluates five state-of-the-art models on 1,498 everyday videos, with annotations for plausible emotions, action tendencies, and world-state constraints.Its evaluation combines automatic perception metrics with human-calibrated VLM/LLM judgments.Source: §1 · p. 2
  • World-state change scores reached only 0.10–0.28, counterfactual emotion sensitivity reached 0.027–0.070, and counterfactual action realization dropped by 41%.Models maintained physical consistency but struggled to infer plausible emotions, realize emotion-driven actions, and generate their consequences.Source: §1 · p. 2

2 Related Work

Related work extends world models from physical dynamics toward human social dynamics, while existing benchmarks still emphasize physical, perceptual, or task-oriented variables. AffectWorld complements those benchmarks by testing emotion-driven continuous future states.

2.1 World Models

  • World models span latent dynamics, observable-space generation, and embodied uses such as data synthesis, policy evaluation, and action planning, with newer work representing social and mental states.Social world models represent intentions and beliefs; mental world models jointly evolve beliefs, goals, emotions, and social norms with the physical world.Source: §2.1 · p. 3

2.2 World Model Benchmarks

  • Existing benchmarks test physical commonsense, perceptual and generative quality, or embodied utility, while AffectWorld evaluates continuous future states driven by emotion.Human-centered evaluations have largely used discrete option selection or textual narratives; AffectWorld adds progressive tasks and an optional counterfactual subtask.Source: §2.2 · p. 3

3 AffectWorld Benchmark

AffectWorld evaluates whether world models predict plausible futures through the linked stages of emotion, action, and world state. It combines three progressive tasks with an optional counterfactual subtask that tests responses to emotional interventions.

3 AffectWorld Benchmark

  • AffectWorld progresses from world-state consistency to emotion-action consistency and causal consistency, using automatic perception metrics and VLM/LLM semantic judgments.The tasks are reported separately to help identify where models fall short.Source: §3 · p. 4

3.1 Data Construction

  • The benchmark contains 1,498 everyday video samples with event and emotion annotations, plausible actions, and constraints on future world states.Samples cover three scene classes across 16 scenarios; annotations use six emotion families and seven action categories.Source: §3.1 · p. 4

3.2 Task 1: World-State Consistency

  • Task 1 measures identity preservation, spatial structure stability, and temporal coherence, combining the three dimensions with a geometric mean.These measures assess continuity of agents and objects, scene layout, and motion over time.Source: §3.2 · pp. 4–5

3.3 Task 2: Emotion-Action Consistency

  • Task 2 assesses plausible event-related emotions and compatible actions, using VLM descriptions and LLM judgments that allow multiple plausible emotions and action tendencies.Action scoring checks whether behavior realizes an action’s function and whether its manner fits the inferred emotion.Source: §3.3 · p. 5

3.4 Task 3: Causal Consistency

  • Task 3 evaluates whether actions produce consistent social and physical consequences against constraints for required, prohibited, and directional world-state changes.Its score also includes a component for whether any world-state change occurs.Source: §3.4 · p. 5

3.5 Optional Subtask: Counterfactual World

  • The counterfactual subtask holds the scene and event context fixed while changing the target emotion, then assesses future plausibility and emotion-specificity across both branches.The global sensitivity score averages across the counterfactual sample triplets.Source: §3.5 · pp. 5–6

4 Experiments

Across five models, the experiments find a gap between maintaining physical world structure and predicting emotion-driven social behavior and consequences. Performance also weakens as social scenes become more complex, and counterfactual emotion changes rarely produce sufficiently different futures.

4.1 Experimental Setup

  • Five models were evaluated on AffectWorld; video-to-video models received history video, while image-to-video models received the first frame and textual scene and event context.Semantic scores used VLM-extracted descriptions and LLM judgments, with human review and calibration.Source: §4.1 · p. 6

4.2 Main Result

  • Models maintain reasonable world-state structure but struggle to infer plausible emotions and realize their behavioral and causal consequences.The paper concludes that current video generation models have basic physical world-modeling capability but have not learned to treat emotion as a causal driver of future states.Source: §4.2 · pp. 6–7

4.3 Performance Across Scene Complexity

  • CogVideoX’s Task 1 score falls from 0.756 in solo scenes to 0.353 in groups, while HunyuanVideo’s Task 2 score drops 27% from solo to group settings.The reported pattern spans tasks: dyadic scores fall 15–20%, and Task 3 is below 0.50 for every model in group scenes.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 Fine-grained Discussion

  • On Task 2, HunyuanVideo scores 0.7496, 40% higher than the best V2V model despite less visual history; authors suggest text-based semantic conditioning grounds emotion-driven inference better than implicit temporal patterns.Event-to-emotion mapping remains the weakest link: HunyuanVideo scores 0.7088 on Memo, while the poorest V2V model scores 0.4155.Source: §4.4 · pp. 7–8
  • All models favor temporal continuity over spatial consistency: Stemp and Sid exceed 0.7, while Sdepth remains below 0.55.The paper notes that proximity, orientation, and positioning carry emotional meaning in social scenes, but models fail to stabilize these relations.Source: §4.4 · pp. 7–8
  • HunyuanVideo reaches 0.5000 on Sdir but only 0.2801 on Smust, indicating stronger directional consistency than generation of specific required outcomes.The authors characterize this gap as broad emotion–behavior associations without fine-grained causal reasoning about concrete world-state changes.Source: §4.4 · p. 8

4.5 Counterfactual emotion sensitivity

  • HunyuanVideo’s counterfactual emotion sensitivity is 0.0701, over tenfold below its Task 2 score of 0.7496.Under counterfactual emotions, action realization drops 41%; weak emotion selectivity means models produce similar actions for joyful and anxious agents.Source: §4.5 · p. 8, Table 2 · p. 8

4.6 Case Study

  • In the case study, Cosmos keeps a downcast girl nearly motionless instead of showing the reference future’s contact with friends, getting up, and meal preparation.The example illustrates limited behavioral progression toward engagement or self-care without explicit action guidance.Source: §4.6 · p. 8, Figure 5 · p. 8

5 Conclusion

AffectWorld assesses emotion-driven future prediction through consistency tasks and counterfactual emotion interventions. Experiments identify persistent weaknesses in generating consequences of emotion-driven actions and responding to emotional changes.

5 Conclusion

  • AffectWorld evaluates world-state consistency, emotion–action consistency, causal consistency, and responses to counterfactual changes in emotion.Its world-state constraints allow multiple plausible futures rather than requiring a single reference trajectory.Source: §5 · pp. 8–9
  • Experiments on five world models reveal persistent weaknesses in generating emotion-driven action consequences and responding to emotional interventions.The benchmark is presented as a basis for measuring progress in modeling human behavior alongside physical dynamics.Source: §5 · p. 9

Limitations

The stated limitations concern the benchmark’s video scope, incomplete observability of emotions and social consequences, and reliance on VLM/LLM judges.

Limitations

  • AffectWorld uses short everyday videos that may not represent diverse cultural contexts or long-term social dynamics.Annotations describe plausible transitions rather than uniquely correct latent states or futures, and semantic evaluation partly relies on VLM/LLM judges that may inherit evaluator biases despite human calibration.Source: Limitations · p. 9

Ethics Statement

The benchmark uses publicly available short-video platform samples and is intended for research on AI modeling of emotions and social dynamics.

Ethics Statement

  • The authors discourage using models trained or evaluated on the benchmark for real-time monitoring, profiling, or applications infringing on privacy or autonomy.They identify surveillance, manipulation, and discrimination as potential misuse risks and call for compliance with applicable regulations and ethical guidelines.Source: Ethics Statement · p. 9

AI Assistance Statement

AI tools assisted with drafting and initial code templates, while the authors developed the paper’s core contributions and reviewed the generated material.

AI Assistance Statement

A Comparison with Existing Benchmarks

AffectWorld is compared with physical-world and affective/social-understanding benchmarks across physical understanding, emotional causality, and generated-content evaluation. It uniquely combines generated-content evaluation with emotion treated as a causal variable.

A Comparison with Existing Benchmarks

  • AffectWorld is the only compared benchmark combining generated-content evaluation with emotion as a causal variable.Physical understanding appears only through constraints that keep generated worlds physically plausible.Source: A Comparison with Existing Benchmarks · p. 11
  • The comparison groups prior benchmarks into physical-world benchmarks and affective or social-understanding benchmarks.The latter may take images, videos, text, or dialogue as inputs.Source: A Comparison with Existing Benchmarks · p. 11
  • Three dimensions distinguish benchmark coverage: physical understanding, emotional causality, and evaluation of generated content.Emotional causality means emotion drives subsequent behavior and world-state change rather than serving only as a recognition or reasoning target.Source: A Comparison with Existing Benchmarks · p. 11

B Data and Annotation Definitions

The paper specifies taxonomies, world-state constraints, and counterfactual annotations for evaluating emotion-driven futures, alongside visual and causal-consistency measures. It also describes the benchmark’s everyday-video scenarios.

B.1 Scene, Emotion, and Action Taxonomies

  • The taxonomy organizes scenes, emotions, and action tendencies into candidate categories for emotionally coupled behaviors.It covers 16 scene sub-categories, 30 emotion sub-categories, and 20 action-tendency sub-categories.Source: §B.1 · p. 11

B.2 World-State Dimensions and Constraints

  • World-state annotations cover physical objects, spatial relations, agent internal states, and social atmosphere, using must, must_not, and directional constraints.Items are limited to changes strongly causally related to the sample’s event and emotion.Source: §B.2 · pp. 11–12

B.3 Counterfactual Subset

  • The counterfactual subset fixes the physical scene and event while replacing the target agent’s emotion, then re-derives actions and world-state consequences.The resulting branches share the same initial state but diverge in emotion and future evolution.Source: §B.3 · p. 12

C Task 1 Detailed Formulation

  • Task 1 measures identity preservation, spatial stability, and temporal coherence using tracked-region features, depth maps, and interpolated-frame comparisons.Identity uses flow-weighted DINOv2 similarity; spatial stability uses temporal depth-map variance; temporal coherence uses SSIM weighted by optical-flow importance.Source: §C.1 · pp. 12–13, §C.2 · p. 13, §C.3 · p. 13

D Task 3 Detailed Formulation

  • Task 3 checks whether must consequences occur, must_not changes are avoided, directional trends are followed, and any world-state change exists.Directional items are averaged as graded evidence rather than required jointly.Source: §D.1 · p. 13, §D.2 · p. 13, §D.3 · p. 13

E Counterfactual Evaluation Detailed Formulation

  • Counterfactual evaluation assesses whether a generation is plausible under its emotion and specific to that condition rather than the other emotion.Emotion selectivity is the relative plausibility drop when an emotion’s annotations are applied to the other-condition video.Source: §E.1 · p. 13, §E.2 · p. 14

F.1 Video Visualization

  • The video overview spans 16 everyday scene categories across solo, dyadic, and group interactions, using short-video-platform samples.Consecutive key frames from initial-state segments capture characteristic visual patterns and emotion-triggering contexts.Source: §F.1 · p. 14

F.2 Duration Analysis

The duration analysis describes source videos and extracted initial-state clips across 1,498 samples, with shorter clips providing context for anticipating emotion-driven transitions.

F.2 Duration Analysis

  • 10.0 seconds is the median source-clip duration across the 1,498 video samples.The mean duration is 9.8±2.3 seconds.Source: §F.2 · p. 14
  • 1.65 seconds is the median duration of extracted state_0 segments.Their mean duration is 1.86±1.05 seconds.Source: §F.2 · p. 14
  • The state_0 segments provide event-background and initial-emotion context for anticipating later actions and world-state effects.The duration distributions are approximately unimodal, balancing contextual information with rapid emotional dynamics.Source: §F.2 · p. 14

G Evaluation Parameters

This section describes evidence-grounded procedures for annotating emotion, action, and world state from video, with temporally grounded outputs and review rules that accommodate plausible emotional futures while filtering or flagging weakly supported samples.

G Evaluation Parameters

  • For Tasks 2 and 3, semantic judgments are repeated three times at temperature 0.3 and median scores are retained; candidate items are shuffled to reduce order effects.Task 2 maps emotion and action-realization levels 0–4 to five scores and coherence levels 0–2 to weights 0, 0.75, and 1; Task 3 scores required consequences, prohibited changes, and directional changes using the stated discrete mappings.Source: G Evaluation Parameters · pp. 14, 16

H.1 Two-Stage Architecture

  • The annotation pipeline uses a VLM to describe visible facts and an LLM to infer structured emotion–world-state transitions, keeping observation distinct from interpretation.This separation makes the evidence for high-level inferences traceable and allows the observational and inferential stages to be audited independently.Source: H Data Annotation Pipeline · p. 16, §H.1 · p. 16

H.2.2 Core Principles

  • Stage 1 descriptions use temporal sequencing and anchors, track a stable main person, preserve visible actions and state changes, and report discontinuities without repairing missing steps or inferring emotion or causality.Descriptions preserve relevant initial states, events, immediate responses, physical cues, object changes, and others’ responses; uncertainty is stated only when it affects interpretation.Source: §H.2.1 · p. 16, §H.2.2 · p. 16

H.2.3 Prompt Framework

  • The Stage 1 prompt requires concrete observables, meaningful temporal markers, stable-person tracking, and explicit handling of identity or object discontinuities while forbidding emotional and causal inference.Its implementation notes say these constraints and object-state tracking help prevent invented explanations and omitted intermediate steps.Source: §H.2.3 · p. 17, §H.2.4 · p. 17

H.3.1 Design Motivation

  • The LLM annotations connect an initial emotional state and triggering event to plausible emotional responses, action tendencies, and atomic world-state changes across plausible futures.The design requires these claims to remain causally consistent and grounded in the VLM’s observable evidence.Source: §H.3.1 · p. 17

H.3.2 Core Principles

  • Samples require an identifiable subject, a visible non-neutral initial emotion, a grounded trigger, a later observable response and world-state change, and a plausible emotional link between trigger and response.A clearly observed facial-expression or vocal-tone change may serve as the sole agent-state atom when it is a distinct response with before-and-after observations of the same subject.Source: §H.3.2 · pp. 17–18
  • Annotations place the initial state and later response in ordered time-anchor intervals, and represent world-state constraints as must, must_not, or directional atoms across four dimensions.Every atom needs before-and-after evidence about the same entity or relation, with the after evidence later than state0; annotators must not invent identities, causality, or off-screen transitions.Source: §H.3.2 · p. 18

H.3.3 Prompt Framework

  • The Stage 2 prompt requires exact emotion and action labels, a visible non-neutral initial emotion, a grounded trigger, plausible emotional influence, and observable later change; it prohibits fabricated affect, events, or outcomes.The prompt requires the world-state constraints to use must, must_not, and directional arrays and the four atom types object_state, spatial_relation, agent_state, and social_atmosphere.Source: §H.3.3 · pp. 18–19
  • Annotations require independently observable before-and-after evidence for the same entity, prohibit invented identities or off-screen transitions, and allow must, must_not, and directional constraints.The criteria exclude camera framing, zoom, and cut-driven visibility as world-state changes.Source: §H.3.3 · p. 19

H.3.4 Implementation Notes

  • Closed vocabularies and explicit atom schemas constrain annotations while admitting multiple plausible futures, and candidate, excluded, or needs_review decisions keep weakly supported samples out of the benchmark or flag them for review.Expanded factual-tier criteria include subtle but clearly observed affective responses when grounding and causal structure remain intact.Source: §H.3.4 · p. 19

H.4 Human Review and Calibration

  • Reviewers checked 300 of 1,498 original samples and 60 of 233 counterfactual cases; semantic rubrics were calibrated on 60 generated videos and judge–human agreement tested on 150 held-out videos.Three annotators rated semantic videos independently, and the calibration and held-out sets used different historical inputs.Source: §H.4 · pp. 19–20
  • Event-Emotion Plausibility had the highest human agreement (α = 0.78) and judge–human correlation (ρ = 0.83), with the lowest MAE (0.09); Causal Consistency scored α = 0.72, ρ = 0.77, and MAE = 0.12.The lower agreement and larger error for Causal Consistency indicate that evaluating world-state consequences is more challenging.Source: Table 6 · p. 20, §H.4 · p. 20
Loading 2610.18305v1…