Source-linked AI summary
Walk through Paintings: Egocentric World Models from Internet Priors
Anurag Bagchi, Zhipeng Bao, Homanga Bharadhwaj, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert
TL;DR
Action-conditioned world models must predict physically correct futures without requiring expensive, broad action-labeled datasets. EgoWM adds lightweight, temporally aligned action conditioning to pretrained video diffusion models, and achieves stronger structural consistency while generalizing across architectures, embodiments, and unseen environments.
Problem
Action-conditioned world models lack scalable, broadly applicable training approaches because paired action-observation data are expensive and existing methods are often domain-specific.
Method
EgoWM converts pretrained video diffusion models into action-conditioned world models by projecting and temporally downsampling actions, then injecting them through the diffusion timestep pathway.
Results
EgoWM improves Structural Consistency Score by up to 65% over prior work and generalizes across diffusion architectures, 3-DoF navigation, 25-DoF humanoid control, and unseen environments.
Takeaways & Limitations
Pretrained Internet-scale video priors can support scalable, controllable world models for navigation, manipulation, and downstream robot planning.
Abstract
from arXiv · showhide
What if a video generation model could not only imagine a plausible future, but the correct one -- accurately reflecting how the world changes with each action? We answer this by presenting the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pre-trained video diffusion model into an action-conditioned world model, enabling precisely controllable future prediction. Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models by injecting appropriately compressed motor commands through lightweight conditioning layers. This allows our model to follow actions faithfully while preserving generalization and realism. Our approach scales naturally across embodiments and action spaces -- from 3-DoF mobile robots to 25-DoF humanoids, where predicting egocentric joint-angle-driven dynamics is substantially more challenging. The model produces coherent rollouts for both navigation and manipulation, requiring only modest fine-tuning. To evaluate physical correctness independent of appearance, we introduce the Structural Consistency Score (SCS), which measures whether stable scene elements evolve consistently with the provided actions. Our method improves SCS by up to 65\% over the prior state of the art, Navigation World Models; applies seamlessly to three different video diffusion model architectures; and effectively utilizes Internet priors to generalize to unseen environments, including navigation and manipulation inside paintings. Finally, we demonstrate the applicability of EgoWM to robotic planning.
1 Introduction
EgoWM repurposes pretrained video diffusion models into action-conditioned world models, addressing the need for scalable controllable prediction from limited action-labeled data. It generalizes across architectures, embodiments, environments, and downstream robotic planning while introducing SCS to evaluate physical consistency.
- World modeling predicts how future visual states evolve in response to actions, supporting navigation, manipulation, and planning in novel settings.
- Existing action-conditioned datasets and models are often expensive, domain-specific, or tied to narrow robot setups, motivating reuse of Internet-scale video priors.
- EgoWM generalizes across video diffusion backbones and embodiments, scaling from 3-DoF navigation to 25-DoF humanoid navigation and manipulation, including unseen painted environments.
- Structural Consistency Score measures whether scene structures evolve consistently with actions, independently of texture and appearance.
- EgoWM converts image-to-video diffusion models into action-conditioned world models without changing their base architectures, using lightweight action conditioning aligned to latent temporal compression.
- EgoWM supports downstream robot trajectory planning in both 3-DoF navigation and 25-DoF manipulation while requiring only modest action-labeled data.
2 Related Work
Prior world-modeling research spans robotics, games, manipulation, and pretrained video-generation models, but commonly remains confined to narrow domains, camera settings, or bespoke architectures.
- Early robotics world models predicted future camera frames from actions and supported control through model-predictive planning or stochastic future prediction.
- Diffusion-based gaming world models preserve visual detail or simulate games in real time, but are trained from scratch on domain-specific data.
- Tabletop manipulation models typically use narrow domains and static exocentric cameras, unlike agents operating from egocentric views while moving through environments.
- Recent methods reuse video-generation pretraining for navigation and manipulation, but differ in conditioning mechanisms and architectural scope.
- Pretrained video diffusion representations have also been adapted beyond synthesis for downstream vision and control tasks, supporting generalization to unseen domains.
3 Method
EgoWM converts pre-trained video diffusion models into action-conditioned world models by aligning compressed action embeddings with latent video dynamics and injecting them through timestep modulation. The design preserves compatibility across architectures and embodiments while enabling fine-grained motion control and structural consistency evaluation.
- 3.1 Preliminaries: Pre-trained video diffusion models generate future frames by iteratively denoising a latent conditioned on an observed frame and optional text.The denoising process uses a timestep embedding that provides the shared control pathway for EgoWM.
- 3.2 Action Conditioning: EgoWM embeds each action vector into a latent representation and downsamples it by the model’s temporal compression factor for alignment with latent video frames.For compression factor k, the action representation has temporal length T/k.
- 3.2 Action Conditioning: EgoWM injects temporally aligned action embeddings through the diffusion timestep-modulation pathway rather than adding model-specific conditioning layers.The action and timestep embeddings are combined before the base model’s block-specific projection.
- 3.2 Action Conditioning: The conditioning framework supports U-Net and DiT backbones without changing their architectures, adapting across temporal resolutions, action dimensionalities, and embodiments.The same design covers 3-DoF navigation and 25-DoF humanoid control; humanoid control additionally uses the initial agent-state embedding to address unseen body parts.
- 3.2 Action Conditioning: The method provides fine-grained motion control in low- and high-DoF settings while enabling extreme generalization.The figure compares generated rollouts using perceptual metrics and SCS, which identifies the physically consistent action trajectory.
4 Structural Consistency Score
Structural Consistency Score (SCS) evaluates whether generated scene structures evolve in accordance with agent actions, complementing appearance-based metrics that may reward visually sharp but physically inconsistent predictions. It tracks passive scene objects over valid frames and averages their mask IoU against ground truth.
- Motivation: Perceptual metrics such as LPIPS and DreamSim can assign similar scores to predictions with different action-following behavior.They emphasize visual fidelity or realism rather than structural alignment in complex scenes.
- Computation: SCS trims frames containing only novel regions, then evaluates passive scene structures whose apparent motion is caused by the agent’s actions.Dense point tracking identifies when initially visible points have left the field of view, after which selected objects are tracked in predicted and ground-truth sequences.
- Computation: SCS averages mask IoU across all evaluated frames and tracked objects, with higher values indicating stronger structural alignment.For each frame and object, the score compares predicted and ground-truth binary masks.
- Interpretation: SCS correlates with action following by measuring structural evolution rather than appearance.It therefore quantifies how faithfully a model predicts the consequences of agent actions.
5 Experiments
EgoWM is evaluated across 3-DoF navigation, 25-DoF humanoid navigation and manipulation, multiple backbones, and out-of-distribution painted environments. It consistently follows actions while preserving realistic structure, with strong gains over NWM and broad embodiment and architecture generalization.
- Navigation Results: All EgoWM variants outperform NWM across evaluation horizons and metrics, with SCS improvements reaching up to +65%.The largest gains occur at greater distances from the initial frame, indicating stronger action alignment.
- Navigation Results: EgoWM produces realistic rollouts that closely follow trajectories, while NWM exhibits noticeable drift.Cosmos and Wan provide sharper predictions, whereas SVD shows mild fidelity loss from chunk-wise autoregressive inference.
- Efficiency: EgoWM achieves up to 6× faster 64-frame inference than NWM while using approximately 8× less training compute.The comparison uses a single A100 GPU on the same 20 RECON samples; EgoWM also uses less action data and produces higher-resolution outputs.
- Humanoid Results: EgoWM scales from 3-DoF control to the 25-DoF humanoid action space without architectural modifications.The higher-dimensional setting is more difficult, but the same conditioning approach supports navigation and manipulation across embodiments.
- Humanoid Results: In 25-DoF manipulation, all variants achieve high SCS and accurately follow commands, although temporal compression affects long-horizon action following.SVD performs better further into the future than Cosmos and Wan, a trend attributed to temporal compression.
- Humanoid Results: EgoWM generates stable humanoid trajectories for navigation, reaching, and grasping while maintaining coherent scene structure and plausible articulation.The predictions remain visually consistent and physically plausible for downstream planning applications.
- Out-of-Distribution Generalization: EgoWM generalizes to navigation and manipulation inside paintings, including distinct trajectories and humanoid apple-picking actions.These results test both 3-DoF navigation and 25-DoF manipulation under a drastic appearance shift.
6 Planning with EgoWM
EgoWM supports goal-conditioned planning by rolling out candidate actions and selecting those whose predicted outcomes match a goal image. This procedure achieves high manipulation success and recovers the target success rate for most evaluated tasks.
- Planning Setup: CEM samples candidate action sequences, rolls them out in EgoWM, and selects actions using similarity between predicted final frames and a goal image.The evaluation covers 3-DoF RECON navigation and 25-DoF humanoid manipulation.
- Navigation Planning: Both EgoWM variants achieve lower average trajectory error in 3-DoF navigation planning over 16-frame trajectories.The reported evaluation covers 4-second RECON trajectories.
- Manipulation Planning: EgoWM-selected trajectories achieve high success across several 25-DoF humanoid manipulation tasks.Success rate is used instead of ATE because multiple trajectories can complete a manipulation task.
- Manipulation Planning: For most manipulation tasks, EgoWM recovers the target success rate of 100% when rolling out ground-truth demonstrations.This result demonstrates the model’s potential for policy evaluation.
7 Conclusion
The paper presents EgoWM as a general framework for converting pre-trained video diffusion models into controllable world models. Its experiments support scalable generalization across embodiments and environments, physically consistent futures, and downstream decision-making.
- Conclusion: EgoWM adds lightweight action conditioning to existing video diffusion architectures and leverages Internet-scale priors.The framework is presented as a route toward scalable, controllable, and generalizable world models.
- Conclusion: Experiments demonstrate generalization across diverse embodiments and environments, including humanoids, while supporting downstream decision-making.The conclusion also highlights physically and semantically consistent futures.
- Conclusion: Structural Consistency Score is introduced to evaluate action following independently of appearance.The metric is positioned as a way to assess whether generated futures remain physically consistent with actions.
A Qualitative Video Results
The qualitative video results examine EgoWM’s zero-shot behavior in paintings and its action following under varied trajectories and temporal compression. They show broad motion control alongside concrete failures in object permanence and some painted-domain grasps.
- Zero-shot Generalization to Paintings: EgoWM generates distinct, motion-consistent navigation trajectories from the same initial painting.The demonstrations include both 3-DoF navigation and 25-DoF manipulation in painted scenes.
- Zero-shot Generalization to Paintings: In paintings, EgoWM can simulate actions such as picking up an apple or pulling a tablecloth while accurately following commanded sequences.Some grasps remain physically inaccurate because painted object interactions are absent from training data.
- Zero-shot Generalization to Paintings: The model accurately follows specified position and velocity trajectories in 3-DoF position-control examples.These examples extend the painted-domain evaluation beyond a single commanded motion type.
- Real-World and Long-Horizon Results: EgoWM maintains distinguishable behaviors under small action variations even with temporally compressed models.The webpage also reports failures when rearranging small blocks, where object permanence is difficult.
B Structural Consistency Evaluation
SCS evaluates whether generated scene structures remain physically aligned with action-conditioned ground truth, rather than merely matching appearance. Its tracking-based computation is robust to severe visual distortion and can be automated without changing method rankings.
- SCS tracks salient scene structures from initial-frame annotations and compares their predicted locations with ground-truth locations across navigation and manipulation.The metric uses SAM2 to track entities such as houses, trees, and robot arms.
- SCS is designed to measure structural alignment independently of rendering fidelity, so distorted but structurally consistent predictions remain assessable.The evaluation adds Gaussian noise and uses SAM2 tracking accuracy to test robustness.
- 0.93 mIoU at PSNR 16.5 shows that SAM2 tracking remains accurate under heavy frame distortion.This supports using SCS to probe structural consistency rather than superficial visual artifacts.
- Automated segmentation and tracking preserve the relative ranking of methods across prediction horizons, removing the need for manual test-sequence annotation.The automated variant supports scaling SCS evaluation to larger datasets.
C Long-Horizon Prediction
EgoWM maintains action-conditioned structural consistency over 64-frame autoregressive rollouts, extending evaluation beyond the 16-frame horizons used in the main tables. Navigation degrades less than the NWM baseline, while manipulation remains largely stable.
- 64-frame autoregressive rollouts evaluate EgoWM at horizons 4× longer than the 16-frame main-table setting.The analysis covers both 3-DoF navigation and 25-DoF manipulation.
- EgoWM’s SVD and DiT navigation variants degrade less significantly over long rollouts than the autoregressive NWM baseline.The comparison uses SCS on long-horizon navigation predictions.
- Manipulation SCS remains largely stable through 64 frames, indicating controllability at horizons relevant for policy learning.
D Compute and Latency
EgoWM improves over NWM while using less action data and substantially less training compute, and it generates at higher native resolution. Its inference is also faster for long rollouts despite the higher resolution.
- EgoWM uses approximately 8× less training compute and less action data than NWM while generating higher-resolution predictions.Main-paper metrics are compared at 224 × 224 after downsampling EgoWM outputs, whose native resolutions are higher.
- EgoWM achieves up to 6× faster 64-frame inference while predicting at up to 2× higher resolution.The latency comparison uses a single A100 GPU and each model’s native resolution.
E Implementation Details
The implementation trains SVD and Cosmos variants with lightweight action-specific optimization and autoregressive or direct multi-frame prediction. Reported failure cases remain concentrated in manipulation object consistency and out-of-distribution scene generalization.
- Implementation Details: SVD and Cosmos are trained on 8 A100 GPUs, using higher learning rates for the newly introduced action projection layers.The backbone learning rates are 1e-5 for SVD and 1e-6 for Cosmos, versus 1e-4 and 1e-5 for their action projections.
- Implementation Details: SVD predicts 8 future frames at 512×512 and rolls out 16 frames autoregressively, while Cosmos and Wan directly predict 16 frames at 480×640.Training and inference use 5 FPS frame subsampling.
- Failure Modes: Manipulation can distort small objects across occlusions, and object permanence remains challenging during complex manipulation.The reported square-block failure is attributed primarily to limitations of the underlying video-generation backbones.
- Failure Modes: Painting generalization can collapse toward real-world scenes or develop structural inconsistencies when rapidly synthesizing unseen regions.These failures can reduce stylistic consistency and weaken representations of people and structures.