Source-linked AI summary
AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization
Yu Li, Menghan Xia, Gongye Liu, Xintao Wang, Conglang Zhang, Lei Ke, Yuxuan Lin, Ruihang Chu, Pengfei Wan, Kun Gai, Yujiu Yang
TL;DR
Interactive world modeling lacks versatile control for embodied first-person use, especially when body motion is only partially visible and local world states must evolve consistently. AnchorWorld uses hybrid-view 3D human-motion control and pose-associated anchor views with textual evolution prompts, and reports improvements over existing methods with validated design components and localized customization capabilities.
Problem
Interactive world models have limited support for natural embodied control and explicit, persistent customization of local states in first-person environments.
Method
AnchorWorld combines 3D human motion and hybrid-view projection-based conditioning with pose-associated anchor views containing appearance, 3D pose, and evolution prompts.
Results
AnchorWorld consistently surpasses existing methods, while experiments and ablations support accurate action control, spatial awareness, scene consistency, and controllable evolution.
Takeaways & Limitations
The framework supports localized world customization, including out-of-sight scene evolution and pose-consistent anchoring under spatial transformations.
Takeaways & Limitations
The current work focuses primarily on short video clips, leaving longer-horizon exploration and real-time environmental state updates as challenges.
Abstract
from arXiv · showhide
Despite being a pivotal frontier, interactive world modeling remains underexplored in terms of the versatile controllability required by practical scenarios. To bridge this gap, we present AnchorWorld, a framework that advances egocentric simulation through enhanced interaction integrity and a flexible mechanism for world customization. First, we utilize 3D human motion as the primary interaction modality. To complement the out-of-view or truncated body parts in egocentric views, we introduce an auxiliary training supervision that incorporates exogenous viewpoints decoupled from the agent's first-person sensorium. It allows the model to observe the agent's full-body positioning relative to the environment, facilitating a more robust spatial grounding of human-world interactions. Furthermore, we propose a simple yet effective mechanism for customizing self-evolving worlds. This is achieved by defining anchor views within a unified world coordinate system, coupled with textual descriptions dictating the dynamic evolution of local scenes. Experimental results show that AnchorWorld significantly outperforms state-of-the-art baselines, while ablation studies validate the effectiveness of our key designs. Notably, our customization scheme exhibits promising spatio-temporal geometric consistency and adheres strictly to the prescribed evolutionary dynamics.
1 Introduction
AnchorWorld addresses the need for egocentric world simulation that responds to human movement while allowing local world states to be specified and evolved. It combines hybrid-view motion supervision with pose-associated anchor views and improves action control, scene consistency, and dynamic evolution across scenarios.
- Interactive egocentric world models must represent how head, body, and coordinated actions affect camera views, navigation, and object interactions.
- Existing methods often use keyboard, camera, or text controls, while egocentric motion learning suffers from sparse supervision because most of the body is outside the predicted frame.
- AnchorWorld combines 3D human motion with camera viewpoints from external or head-mounted views for hybrid-view action conditioning.
- Pose-associated anchor views specify local appearance, grounded 3D pose, and textual evolution dynamics for world customization.
- AnchorWorld improves over adapted baselines on action accuracy, scene consistency, and dynamic evolution across egocentric, synthetic UE, and captured real-world scenarios.
2 Related Work
Prior interactive world-model research controls simulated environments through keyboard-and-mouse operations or text prompts. Related scene-consistent video work explores camera-trajectory synthesis and image-based scene representations for camera control.
- Interactive World Models: Early interactive world models use keyboard and mouse operations to control viewpoints and navigate simulated worlds.
- Interactive World Models: Another research line uses text prompts to trigger specific world events and drive environmental transitions.
- Scene-Consistent Video Generation: ReCamMaster synthesizes novel camera trajectories through source-video conditioning and constructs paired trajectories using synthetic Unreal Engine data.
- Scene-Consistent Video Generation: CineScene represents scenes with dense image sequences captured at regular angular intervals and uses implicit 3D features for camera-controlled scene understanding.
3 Method
AnchorWorld combines hybrid-view human-action control with pose-associated anchor views and localized text prompts to synthesize spatially grounded, temporally evolvable egocentric worlds. Its progressive training strategy builds action control before static customization and dynamic evolution.
- Framework Overview: AnchorWorld conditions egocentric video synthesis on embodied human motion and pose-associated anchor views for navigation, interaction, and world customization.Human motion is represented with SMPL-X body actions, while anchor views provide RGB images, 6-DoF poses, and evolution prompts.
- Hybrid-View Human Action Control: Hybrid-view supervision combines full-body motion with camera trajectories so third-person data supplies complete motion context for first-person action control.The projection-based formulation supports joint training on third-person and first-person-view data and improves spatial grounding between motion and scene responses.
- Hybrid-View Human Action Control: Spatial pose attention concatenates video, motion, and camera tokens, processes them jointly, and retains only updated video features after discarding auxiliary pose tokens.Motion and camera encoders temporally align their embeddings with the video latents before spatial self-attention.
- Evolvable Anchor-View Customization: Anchor-view customization uses in-context image conditioning, distinct positional encodings, and pose injection to associate generated trajectories with localized world constraints.Anchor images guide synthesis without architectural changes to the base model, while 3D poses distinguish views at different world locations.
- Progressive Multi-Stage Training Strategy: Progressive training proceeds from third-person action learning and first-person adaptation to static anchor-view conditioning and dynamic prompt-driven evolution.The four stages establish action controllability before adding consistent roaming and text-driven local state changes.
- Evolvable Anchor-View Customization: Masked cross-attention restricts each evolution prompt to generated video tokens and its corresponding anchor-view tokens, supporting local temporal changes with reduced cross-view interference.This mechanism provides anchor-specific text control for dynamic scene evolution.
- Qualitative Results: Qualitative comparisons report better egocentric action control, scene consistency under large viewpoint changes, and dynamic scene evolution than the compared methods.The comparison visualizes masked human-action regions and 3D anchor-view poses; CaM-Ego lacks body-motion input, while PlayerOne-Scene uses part-wise control.
4 Experimental Results
Experiments evaluate AnchorWorld across egocentric, synthetic UE, and real-world settings, covering action control, scene consistency, dynamic evolution, and ablations. Results show strong performance and generalization, while targeted studies support the proposed design choices.
- Experiment Settings: Evaluation reports action accuracy, scene consistency, video quality, and, for dynamic scenes, text alignment with evolution prompts.
- Quantitative Results: AnchorWorld achieves the best scene consistency, camera accuracy, and text alignment across test scenarios while maintaining comparable visual quality.
- Qualitative Results: Qualitative comparisons show stronger egocentric motion control, scene consistency under large viewpoint changes, and prompt-driven dynamic evolution than competing methods.
- Generalization: The model generalizes to out-of-distribution UE and real-world scenes with limited or no overlap between anchor and initial ego views.
- Ablation Studies: Removing third-person training, projection-based control, anchor-view pose, or anchor-view RoPE weakens action control or scene consistency.
- Dynamic Evolution: Earlier viewpoint changes reveal intermediate states, whereas delayed changes reveal later states, demonstrating timing-sensitive out-of-sight scene evolution.
5 Conclusion and Limitations
AnchorWorld integrates embodied action control with localized, customizable world-state evolution. The paper reports consistent improvements and identifies limitations in long-term exploration, open-world generalization, and diverse dynamic scenarios.
- Conclusion: AnchorWorld combines natural embodied action control with localized world-state customization.
- Conclusion: Third-person videos provide interaction context and complete human-motion supervision, while anchor views and text specify spatial priors and local evolution.
- Conclusion: Extensive experiments surpass existing methods, and ablations validate each key design.
- Limitations: The stated limitations are long-term exploration, open-world generalization, and diverse dynamic scenario modeling.
A Implementation Details
Implementation uses progressive training with multiple data stages, synchronized cross-view inputs, pose and anchor-view conditioning, and text-based evolution prompts. The setup also addresses camera differences and limitations in egocentric pose estimation.
- Training Strategy: The progressive training procedure is organized into four stages with increasing emphasis on egocentric control, viewpoint changes, and static-scene customization.
- Training Data: Stage I uses 200K curated real action videos and 101K synthetic UE videos, followed by 100K egocentric samples in Stage II.
- Optimization: All videos are processed at 480p, with training conducted on 16 NVIDIA GPUs using batch size 16 and learning rate 1×10−4.
- Data Processing: Egocentric datasets provide synchronized third-person and first-person views, although anchor images may contain the first-person player.
- Data Processing: Third-person and first-person videos may have color discrepancies, but the model uses anchor-view scene information while preserving the initial ego-view color tone.
- Pose Processing: 3D human motion is estimated from third-person views and canonicalized with 22 major body joints, excluding hand poses because current egocentric estimation is unreliable.
- Evolution Prompts: Evolution prompts are annotated with Qwen3-VL-32B-Instruct using designed prompt templates.
B Limitation
AnchorWorld is limited mainly by short-horizon, constrained-scenario training data and current egocentric data coverage. The authors identify longer exploration, open-world environments, and more diverse dynamics as future directions.
- Long-Term Exploration: The framework primarily targets short video clips rather than long-horizon exploration and interaction.Future real-time autoregressive interaction would require updating environment states during continuous embodied interaction and stronger long-term memory.
- Open World: Training data focuses on a constrained set of scenarios, limiting the breadth of environments represented.The authors propose collecting open-world data to support broader environments and longer-horizon exploration.
- Diverse Dynamic Scenarios: Current egocentric training data mainly supports human-related activities and globally consistent evolution descriptions across anchor views.The authors use t1 = · · · = tn and identify diverse scenarios and anchor-specific dynamic controls as future extensions.
C Additional Analyses and Results
Additional analyses show that AnchorWorld supports prompt-driven scene evolution, robust action control in real-world settings, and generalization across anchor-view configurations. Scene-coherence tests further show sensitivity to incompatible appearances and spatially inconsistent geometry.
- Dynamic Evolution: Modifying the evolution prompt produces different dynamic scene evolutions and supports anchor-specific evolution descriptions.This analysis demonstrates flexible user control over prescribed scene dynamics.
- Egocentric Action Control: The method outperforms the compared controls in qualitative body-motion accuracy, while PlayerOne is inaccurate and CaM-Ego supports only viewpoint control.Figure 9 covers in-domain comparisons, and Figure 10 extends visualization to real-world scenarios.
- Ablation: Removing third-person motion knowledge or using non-projection-based control reduces body-motion control accuracy.The ablation isolates both third-person supervision and projection-based action conditioning as effective design choices.
- Egocentric Action Control: AnchorWorld generates stable real-world results for diverse body motions, including pouring water, squatting, jumping, and walking upstairs.The comparison is presented as egocentric action control in real-world scenes.
- Scene Coherence: Scene coherence can fail when the anchor image is stylistically incompatible with the ego-view or when anchor and human poses mirror an inconsistent world geometry.These tests indicate that generated videos require a continuous, complete, and spatially consistent world representation.
C.5 Exocentric Action Control
Exocentric action-control analyses evaluate pose representation and injection strategy while visualizations examine generalization across anchor-view settings and scene transformations. Spatial pose attention performs best, and coherence depends on consistent world geometry.
- Pose Injection: Spatial pose attention achieves the best performance among tested pose-condition injection strategies.It explicitly aligns video tokens with pose tokens and removes pose tokens after attention to address their distribution gap with VAE latents.
- View Generalization: AnchorWorld generates stable results with non-overlapping viewpoints and in both multi-anchor-view and single-anchor-view settings.Additional coherence visualizations show that style mismatch or mirrored poses can still produce incoherent results when world geometry is inconsistent.
C.6 Additional Quantitative Results
Additional quantitative analyses cover anchor-view counts, fine-grained VBench dimensions, and third-person action-control metrics. The section also reports visual failure cases involving fine texture preservation and blur.
- Anchor-View Ablation: Table 5 evaluates how the number of anchor views affects scene-consistency performance.The analysis studies whether additional views surrounding a world scene improve consistency.
- Metrics: Table 4 reports WA-MPJPE and PA-MPJPE for third-person action-control ablations, with lower values indicating better performance.Table 6 breaks down average VBench results into subject, background, temporal, motion, imaging, and aesthetic dimensions.
- Failure Cases: Complex local structures can yield inconsistent fine-grained textures, while rapid viewpoint changes and blurry training frames can produce blurry generation artifacts.These are the reported failure cases attributed to base-model capacity and egocentric-video characteristics.
D Failure Cases
AnchorWorld has two documented failure cases: inconsistent fine-grained scene details and blurry generated results, particularly in rapid-motion settings.
- Complex structures and rich textures can produce inconsistent fine-grained scene details.The authors attribute this limitation largely to the base model’s capacity and latent spatial compression.
- A spatial downsampling factor of 16 in the Wan TI2V 2.2 5B VAE can lose fine-detail information.
- Rapid viewpoint changes in first-person training videos can lead to motion blur in generated results.
- Generated hands may exhibit degraded visual quality because of base-model limitations and fast motion in first-person data.