PaperlayerView exact revision on arXiv

Source-linked AI summary

ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction

Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li

arXiv:2511.20937v1cs.AIcs.CLcs.CVcs.RO

TL;DR

How can embodied cognition be measured in VLMs trained largely without embodied interaction? ENACT frames evaluation as egocentric forward and inverse world-modeling tasks, and finds that VLMs trail humans by a gap that widens with interaction horizon, alongside task and camera-related biases.

  • Problem

    How to measure embodied cognition in largely disembodied VLMs remains unclear, and a unified objective coupling egocentric perception with everyday embodied interactions is missing.

    Source: §1 · p. 2

  • Method

    ENACT evaluates egocentric world modeling as forward observation-ordering and inverse action-ordering tasks, with actions represented as scene-graph changes.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    VLMs trail humans by a gap that widens with horizon, perform better on inverse than forward tasks, and show right-handed and human-like-camera biases.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ENACT assesses models' understanding of environmental dynamics and the consequences of actions through egocentric interaction.

    Source: §5 · p. 13

    The diagnostic tasks are not exhaustive, and computational cost restricted in-depth ablations to a representative subset of models and data.

    Source: §5 · p. 13

Abstract

from arXiv · show

Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.

1. Introduction

Embodied cognition is grounded in sensorimotor interaction, yet how to measure its emergence in largely disembodied VLMs remains unclear. ENACT evaluates it through egocentric world-modeling tasks and finds that VLM performance trails humans, with the gap widening over longer horizons.

Motivation

  • A unified objective coupling egocentric perception with embodied interaction in everyday activities is missing, while prior work and taxonomies cover partial or subjective assessments.Prior work examines static spatial perception, linguistic reasoning and primitive-object interactions.Source: §1 · p. 2

Benchmark

  • ENACT evaluates embodied cognition through forward and inverse sequence-reordering VQA tasks that test how actions transform egocentric observations.The tasks target affordance recognition, action-effect reasoning, embodied awareness and long-horizon reasoning under partial observability, without requiring photorealistic video prediction.Source: §1 · p. 2

Data generation

  • The scalable simulation pipeline uses physics-based scene states to generate key-frame trajectories and QA pairs, including up to millions of QAs from a single episode.The paper reports a dataset of 8,972 QAs from diverse, long-horizon everyday interactions.Source: §1 · pp. 2–3

Findings

  • VLMs trail humans, with the gap widening as horizons grow; models do better on inverse than forward tasks and show right-handed and human-like-camera biases.The reported camera-related degradation occurs with non-human-eye-like intrinsics or viewpoints.Source: §1 · p. 3

2. ENACT: Egocentric Interactive Embodied Cognition Test

ENACT represents egocentric interaction as a POMDP over symbolic scene graphs, RGB observations and visible state-change actions. It constructs validated key-frame trajectories from simulation data, turns them into forward and inverse reordering questions, and evaluates exact and pairwise ordering.

2.1. Problem Formulation

  • The benchmark models scene graphs as states, egocentric RGB images as observations and visible scene-graph changes as actions; forward tasks reorder observations, while inverse tasks reorder actions.Each sampled key-frame trajectory is treated as a self-contained POMDP instance.Source: §2.1 · pp. 3–4

2.2. Key-Frame Trajectories Synthesis for Scalable Data Generation

  • Key frames mark abstract state changes; sampled trajectories require visible nonempty state changes, and their reordering questions test long-horizon action reasoning without photorealistic video prediction.The segmentation removes stretches without semantic changes, while trajectory validation permits transient occlusion during object transitions.Source: §2.2 · pp. 4–5

2.3. Dataset Overview and Evaluation Design

  • The benchmark samples 8,972 questions across 29 BEHAVIOR activities and step lengths 3–10, scoring exact ordering with Task accuracy and adjacent-pair ordering with Pairwise accuracy.An online verifier accepts any predicted permutation consistent with the input constraints.Source: §2.3 · p. 5

3. Experiments and Analysis

ENACT evaluations show that VLMs struggle with long-horizon egocentric world modeling, with performance below humans and consistently stronger on inverse than forward tasks. Tests also reveal sensitivity to human-like camera views, right-hand preference, robustness to rendering and robot-appearance changes, and errors dominated by omissions and hallucinations.

3.1. World Modeling as a Proxy for Evaluating Embodied Cognition

  • Human performance exceeds every evaluated VLM; accuracy falls with horizon, and inverse world modeling consistently outperforms forward, with the margin widening as tasks lengthen.GPT-5 and Gemini 2.5 Pro are comparable to humans only at L=3; tasks at L≥8 challenge even the strongest models.Source: §3.1 · p. 6

3.2. Sensitivity to Image Realism

  • Real-world evaluation mirrors simulator trends, while rendering variations produce no significant performance changes across tested settings.InternVL3.5-241B-A28B was evaluated on 960 real-world QA pairs; tested rendering settings had p≥0.2 versus baseline.Source: §3.2 · pp. 7–8

3.3. Do VLMs Exhibit Anthropocentric Bias on Human Vision?

  • Aperture 60, 80, and fisheye views significantly reduce performance, while raising camera height significantly lowers forward-task accuracy by Δ = −0.13.Aperture 30 and the tested low camera height were not significantly different from baseline.Source: §3.3 · p. 9

3.4. Do VLMs Exhibit Anthropocentric Bias on Embodiment?

  • Robot appearance changes have little effect, but right-hand predicates outperform left-hand predicates across forward and inverse metrics.For forward tasks, 9.38% of true left-hand changes were attributed to the right hand, versus 4.67% of right-hand changes attributed to the left.Source: §3.4 · p. 10

3.5.1. Experimental Setup

  • The error framework compares predicted and ground-truth state differences, classifying mismatches as entity substitution, polarity inversion, predicate substitution, hallucination, or omission.Correct matches are also counted, and state differences are treated as independent events when counts are aggregated.Source: §3.5.1 · p. 11

3.5.2. Error Distribution Analysis

  • Omission and hallucination account for 81% of forward failures and nearly 84% of inverse failures, making them the dominant error types.Forward errors are 43.9% hallucination and 37.1% omission; inverse errors are 41.8% each. Other reported categories include polarity inversion (12.4% forward, 9.2% inverse), entity substitution (6.3% vs. 5.4%), and predicate substitution (0.3% vs. 1.9%).Source: §3.5.2 · pp. 11–12, Figure 6 · p. 11

4. Related Work

Prior work examines embodied cognition through separate perceptual, linguistic, interaction, and world-modeling lenses, but lacks a unified objective for egocentric interaction. ENACT positions transition modeling over interleaved observations and actions as that evaluation lens.

Embodied Cognition

  • Prior work spans static spatial perception, linguistic reasoning and planning, primitive-object interactions, and subjective capability taxonomies, leaving a unified objective for egocentric everyday interaction missing.The section frames ENACT’s lens as jointly connecting spatial perception, physical interaction, and linguistic abstraction.Source: §4 · p. 12

World Modeling

  • World-model research often emphasizes generative fidelity or long-horizon prediction, while related benchmarks cover control, non-interactive scene understanding, temporal ordering, or short-horizon video modeling.ENACT instead studies forward and inverse ordering with an explicitly defined action space and scalable construction.Source: §4 · p. 12

VLMs in Embodied AI

  • ENACT evaluates forward and inverse world modeling over interleaved observation-action sequences, with an explicit scene-graph action space for long-range interactive reasoning.This contrasts with GVL’s reordering of observation sequences for value estimation.Source: §4 · p. 12

5. Conclusions and Limitations

ENACT evaluates embodied cognition through forward and inverse world modeling from egocentric interaction, and experiments find VLMs trail humans by more as interaction horizons grow. Its scope is limited by non-exhaustive diagnostics, selective ablations, and the exclusion of video generative models.

Conclusions

  • VLMs show a significant performance gap from humans that widens with horizon, perform better on inverse than forward modeling, and exhibit handedness and camera-perspective biases.Error analysis identifies omission and hallucination of state changes as the primary reasoning failures.Source: §5 · p. 13
  • ENACT casts forward modeling as ordering future states from actions and inverse modeling as inferring ordered actions from state changes.The benchmark is grounded in a POMDP and assesses understanding of environmental dynamics and action consequences.Source: §5 · p. 13

Limitations

  • The diagnostic tasks are non-exhaustive, ablations cover only representative models and data, and video generative models are not evaluated.The authors cite evaluation cost, physical inconsistency in generated rollouts, and difficulty designing fair metrics as reasons for the video-model scope boundary.Source: §5 · p. 13
  • The authors present ENACT as an extensible framework for broader embodied settings and note that larger evaluations could help generalize findings.They also do not explore VLM fine-tuning in this work.Source: §5 · p. 13

Ethics Statement

ENACT uses simulator-generated data to avoid privacy risks associated with real-world human data; human evaluation annotators were compensated above local minimum wage. The authors also identify limits to simulator realism and potential costs and dual-use concerns.

Ethics Statement

  • The simulator-generated benchmark contains no human subjects or personally identifiable information, avoiding privacy risks associated with real-world human data.Source: Ethics Statement · p. 13
  • Human annotators were paid at rates significantly above their local minimum wage and were not exposed to sensitive content.Source: Ethics Statement · p. 13
  • The authors note that simulator limits may introduce bias and constrain generalizability, while evaluated large-scale models carry computational and environmental costs and potential dual-use applications.Source: Ethics Statement · p. 13

Reproducibility Statement

The paper provides its codebase and dataset and documents notation, key-frame segmentation, and trajectory synthesis. Its pipeline filters scene changes and samples valid fixed-length trajectories from segmented frames.

Reproducibility Statement

  • The public repository includes data-generation and evaluation scripts, an automated verifier, prompts, and code for main experiments, ablations, and human evaluation; the full ENACT dataset is also available.Source: Reproducibility Statement · p. 14

A.1. Notations

  • The notation section collects the paper’s symbols in two tables.Table 5 is captioned as notation used throughout the paper.Source: §A.1 · p. 22, Table 5 · p. 22

A.2.1. Segmented Frames with Abstract State Changes

  • Key-frame segmentation requires a scene change to persist for at least 40 frames, suppresses minor oscillations, and rejects near-duplicate changes using a cosine-similarity threshold of 0.97.The 40-frame threshold is tunable; the paper describes these checks as balancing frame quantity with semantic significance and visual clarity.Source: §A.2.1 · p. 23

A.2.2. Key-Frame Trajectories Synthesis

  • KFTS samples fixed-length trajectories from segmented frames by representing valid, visible transitions as edges in a directed acyclic graph.A transition is valid when the state change is semantically meaningful and visually verifiable.Source: §A.2.2 · p. 23
  • Dynamic programming counts valid trajectories by length and ending frame, and weighted backtracking samples uniformly across valid paths.The method is polynomial in the number of frames and trajectory length, rather than a brute-force combinatorial search.Source: §A.2.2 · pp. 23, 25
  • The symbolic scene-graph abstraction targets semantic changes aligned with VLMs and uses predicates grounded in practical robotics applications, while omitting fine-grained low-level dynamics.The paper points to real-world experiments and BEHAVIOR goal conditions as evidence of the predicates’ practical relevance.Source: §A.2.2 · pp. 25–26

A.3. Dataset Statistics and Evaluation Design

ENACT samples multi-step egocentric trajectories and evaluates them through forward and inverse world-modeling sequence tasks. Its evaluation combines semantic transition matching with exact and graded ordering metrics.

A.3.1. Dataset Statistics

  • The benchmark samples a near-uniform number of problems for each trajectory length from 3 to 10 steps across both task types.The abstract state changes use 11 symbolic predicate classes.Source: §A.3.1 · pp. 26–27, Figure 9 · p. 26

A.3.2. Evaluation Design

  • Actions are grounded in symbolic scene-graph changes, and evaluation compares predicted transitions with reference changes using semantic subset rules.Forward matching requires reference-visible changes to be covered; inverse matching allows a predicted action to be a subset of the full reference transition.Source: §A.3.2 · pp. 27–28
  • Task Accuracy scores full-prediction acceptance, while Pairwise Accuracy gives stepwise credit and uses monotone alignment when prediction and reference lengths differ.Accepted predictions receive Pairwise Accuracy of 1; length-mismatched predictions can still earn pairwise credit.Source: §A.3.2 · pp. 27–28

A.3.3. ENACT Examples

  • The benchmark presents three-step examples of forward and inverse world modeling.Forward tasks order observations given actions; inverse tasks order actions given observations.Source: Figure 10 · p. 28

B.1.1. Annotation Interface & Human Performance Evaluation

  • Three trained annotators completed the same tasks and prompts as the VLMs to provide an empirical human-performance comparison.Separate interfaces support ordering future states in forward tasks and selecting actions for observed transitions in inverse tasks.Source: §B.1.1 · p. 28

B.1.2. Inter-Annotator Agreement Analysis

The authors cross-annotated a balanced sample to assess label reliability, then report agreement statistics. The model evaluation used standardized prompts and input formatting.

B.1.2. Inter-Annotator Agreement Analysis

  • The cross-annotation protocol sampled five questions per annotator for each task type and step length, yielding 240 questions independently labeled by the other two annotators.This design produced three independent labels per question.Source: §B.1.2 · pp. 28, 33
  • Overall Krippendorff’s Alpha was 0.8320, with a 95% bootstrap confidence interval of [0.7879, 0.8682].The authors characterize alpha above 0.80 as high reliability.Source: §B.1.2 · p. 33
  • Annotators agreed perfectly on 184 of 240 questions, an agreement rate of 76.67%.Source: §B.1.2 · p. 33

B.2.1. Experimental Setup

  • Models received a unified prompt per task type, 512 × 512 input images, temperature-zero decoding, and instructions to return a Python-list permutation of indices.The appendix identifies Tables 9 and 10 as reporting Task Accuracy and Pairwise Accuracy, respectively.Source: §B.2.1 · p. 34, §B.2.2 · p. 36, Table 9 · p. 36

B.2.3. Contact Experiment

The supplementary experiments test whether ENACT’s findings depend on key-frame criteria, action encoding, or semantic-verifier partial credit. Their reported patterns remain consistent across the tested variants.

B.2.3. Contact Experiment

  • Adding binary contact changes to key-frame selection preserves the inverse-over-forward advantage and the decline in accuracy with longer interaction sequences.The authors report that the qualitative trends remain the same as in the main results.Source: §B.2.3 · p. 36, Table 11 · p. 37

B.2.4. How Does Action Representation Affect VLMs’ Performance?

  • Across natural-language, symbolic-predicate, and emoji action encodings, inverse modeling remains stronger than forward modeling while both degrade with longer horizons.The comparison uses a 2,304-QA subset and InternVL3.5–241B; absolute accuracies vary slightly across encodings.Source: Table 12 · p. 37, §B.2.4 · p. 38

B.2.5. How Often Do Accepted Predictions Omit Parts of Transitions?

  • Strict-subset omissions occur in about 1–2% of accepted predictions at the data level, versus around 14–15% for models and 3.6% for humans at the pair level.The verifier grants partial credit when predicted predicates are a non-contradictory subset of the ground-truth transition.Source: §B.2.5 · p. 38, Table 13 · p. 38

B.3. Additional Ablation Experiments Common Setup

The appendix describes controlled checks of rendering, camera, and robot-appearance conditions, alongside robustness and error-analysis procedures. The supplied passages detail these setups and report that the main trends persist under predicate-delta perturbations and that handedness errors are asymmetric.

B.3. Additional Ablation Experiments Common Setup

  • The ablations use GPT-5 mini and InternVL3.5-241B-A28B, with variant performance changes assessed against a baseline using Welch’s t-test.The authors treat p < 0.05 as statistically significant and |Δ| < 0.05 as a small change.Source: §B.3 · p. 39

B.4. Sensitivity to Image Realism

  • Image-realism experiments compare realistic generated frames, path tracing, baseline ray tracing, and ray tracing with effects disabled.The generated-frame pipeline aims to preserve trajectory-consistent content and style, while prompt rules address image-generation instability around grippers and actions.Source: Figure 20 · p. 41, §B.4.1 · p. 41, §B.4.2 · p. 42

B.4.4. Robustness to Noise in Predicate Deltas

  • On a 2,304-QA subset, random perturbations to symbolic predicate deltas leave the inverse-over-forward pattern and horizon-related decline unchanged.The robustness study evaluates InternVL3.5–241B across three random seeds and reports small standard deviations across horizons.Source: §B.4.4 · p. 43, Table 14 · p. 43

B.5.2. Fisheye Lens Setup

  • The camera-configuration experiments also include a fisheyePolynomial lens and vary camera height around the 1.75 m default.The high and low camera settings are +0.5 m and −0.25 m from default, respectively.Source: §B.5.2 · p. 44, §B.5.3 · p. 44

B.5.1. Camera Aperture Setup

B.6.1. Robot Appearance

B.6.2. Handedness

  • For both task types, right-hand precision and recall exceed left-hand scores, and left-to-right mixing exceeds mixing in the reverse direction.Source: §B.6.2 · pp. 45, 47

C.1. Methodology for Error Calculations

  • The error analysis represents state changes as atomic scene-graph components and classifies unmatched reference components as omissions and unmatched predictions as hallucinations.Components include edge changes, node changes, and node-state transitions; semantic labels group errors into spatial, functional, material, or agent-interaction categories.Source: §C.1 · pp. 47, 48, 50

C.2. Structural Error Analysis

The analysis compares structural error patterns in forward and inverse tasks across several VLMs and human predictions.

C.2. Structural Error Analysis

  • Structural error patterns are compared between forward and inverse tasks.Source: §C.2 · p. 50
  • The comparison includes Gemini-2.5 Pro and GPT-5.Source: §C.2 · p. 50
  • GPT-5 mini, InternVL-3.5-241B-A28B, and human predictions are also included.Source: §C.2 · p. 50

C.3. Semantic Error Analysis

The semantic error analysis finds that errors cluster around spatial relations and agent interactions, with their prevalence differing between forward and inverse tasks. The paper also reports using LLMs as auxiliary writing and literature-review tools, with human authors responsible for the final content.

C.3. Semantic Error Analysis

  • Errors across systems concentrate in Spatial Relations and Agent Interaction, reflecting difficulty reasoning about object positions and agent actions.Source: §C.3 · p. 50, Figure 48 · p. 55
  • Spatial-relations errors are more common in forward tasks, while agent-interaction errors are higher in inverse tasks.Source: §C.3 · p. 50
  • Structural error categories include polarity inversion, predicate substitution, entity substitution, omission, and hallucination.Source: §C.3 · p. 58

D. The Use of Large Language Models

Loading 2511.20937v1…