Source-linked AI summary
AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization
Cheng Chen, Jerry Bai, Jiacheng Wei, Boyu Chen, Xiaoji Zheng, Fan Wu, Minghao Yang, Tianrun Chen, Ruibo Li, Xiaoyu Yue, Xiaoyang Guo, Yixiao Ge, Guosheng Lin, Fayao Liu
TL;DR
Generalizable manipulation needs scalable contact-rich experience across robot bodies, viewpoints, and scenes, but human videos are fixed to narrow configurations. AnyWorld factorizes action, camera, and embodiment to recompose unpaired human interactions into robot-native rollouts; experiments report controllable recomposition and improved VLA adaptation, while the current method remains limited by missing tactile feedback and evaluation across only three embodiments.
Problem
Scalable robot experience remains difficult to collect across diverse scenes and embodiments, while each human recording captures only a narrow configuration.
Method
AnyWorld factorizes interactions into action, camera, and embodiment conditions, then uses human pretraining and unpaired mixed-embodiment fine-tuning to generate target-robot rollouts.
Results
AnyWorld supports controllable recomposition across bodies, viewpoints, and scenes and improves downstream VLA adaptation in RoboCasa GR1 and real-robot IRON experiments.
Takeaways & Limitations
Generated robot-domain experience can support target-embodiment adaptation, including interventions addressing a spurious completion prior and language-grounded spatial target selection.
Takeaways & Limitations
The method cannot fully capture tactile feedback, depends on reliable action-camera extraction, and has been evaluated across only three embodiments.
Abstract
from arXiv · showhide
Collecting contact-rich robot experiences at scale remains a major bottleneck for generalizable manipulation. Beyond data quantity, robot learning also requires diverse experiences across embodiments, viewpoints, and scenes. Human egocentric videos provide abundant physical interactions, but each video captures only a narrow slice of experience under a single body, camera trajectory, and environment. We propose AnyWorld, a cross-embodiment world modeling framework that expands a single human interaction into diverse robot-native rollouts without paired human-robot demonstrations. Our model factorizes an interaction into action, camera, and embodiment: action controls capture the motion structure, camera controls specify viewpoint evolution, and the target embodiment context defines the acting body and its interaction geometry. This formulation enables independent recomposition of embodiment, viewpoint, and scene factors, allowing a single model to generate many robot-domain experiences while preserving the underlying dynamics and object interactions. We train the model with large-scale human interaction pretraining followed by mixed-embodiment fine-tuning. Experiments show that our model supports controllable recomposition across embodiments, viewpoints, and scenes, and we further demonstrate that the generated data can improve manipulation performance on the RoboCasa GR1 tabletop benchmark and a real IRON humanoid robot. Beyond aggregate gains, we test whether unpaired human experience can be recomposed into robot-native video-action pairs that target a policy gap. Controlled IRON interventions correct a spurious completion prior and establish language-grounded spatial target selection; an action-only counterfactual intervention fails to learn the latter reliably, showing that both action calibration and visual recomposition are necessary.
REFERENCE VIDEOS
The reference materials identify action, scene, and embodiment factors, alongside repeated camera poses, as the visual components of the interaction setup.
- Action, Scene, and Embodiment are the named factors in the reference representation.
- The reference sequence includes multiple camera-pose instances.
- Figure 1 describes transforming human egocentric interactions into robot-native rollouts across embodiments, viewpoints, and scene configurations.
1 Introduction
The introduction frames scalable, diverse robot experience as a central challenge and proposes AnyWorld to recompose human interactions into robot-native rollouts without paired human–robot demonstrations.
- 19: collecting contact-rich robot experience at scale remains a major bottleneck for generalizable manipulation.
- Human egocentric videos offer abundant contact-rich interactions, but each recording remains tied to one embodiment, viewpoint, and environment.
- AnyWorld factorizes interactions into action, camera, and embodiment conditions for recomposing human motion under target robot bodies and scenes.
- The model is pretrained on human interactions and fine-tuned on mixed-embodiment data to support combinations of body, viewpoint, and scene recomposition.
- Experiments report controllable recomposition and improved downstream VLA adaptation in RoboCasa GR1 and real-robot IRON settings.
2 Related Works
Related work spans robotic world models and the use of human egocentric videos for scalable cross-embodiment robot learning.
- Robotic world models: Robotic world models predict visual or latent state evolution under actions and can provide imagined rollouts for policy learning or improvement.
- AnyWorld is positioned at the intersection of action-conditioned world modeling and scalable human-video-based cross-embodiment transfer.
- Human egocentric videos and cross-embodiment transfer: Human egocentric videos have been used for visual representations, rewards, appearance-gap reduction, latent actions, physical tokens, and VLA-related training.
3 Method
AnyWorld generates target-robot egocentric rollouts by independently controlling action, camera, and embodiment–scene context, using unpaired human and robot training data.
- Problem formulation: Zero-shot recomposition extracts action and camera controls from a human video and conditions generation on a target embodiment context.
- Action-camera-embodiment factorization: The factorization separates performed motion, viewpoint evolution, and the body performing the interaction.
- Action control: Pixel-space skeleton control specifies where motion occurs while remaining more embodiment-agnostic than joint commands or robot-specific action vectors.
- Camera control: Camera intrinsics and extrinsics are converted into Plücker ray embeddings to condition viewpoint geometry and separate actor motion from camera motion.
- Embodiment conditioning: The embodiment is specified by an initial frame and text tag encoding body appearance, scene context, object layout, and interaction geometry.
- Training: Training combines large-scale EgoDex human-video pretraining with unpaired multiembodiment fine-tuning in a shared factorized format.
- Inference-time recomposition: At inference, editing the initial frame or changing the camera sequence recomposes the same action under new bodies, scenes, or viewpoints.
- Joint visual-action recomposition: Downstream transfer combines re-embodied observations with morphology-calibrated robot actions, because visual recomposition grounds actions in the target robot-native state.
4 Experiments
AnyWorld is evaluated for controllable recomposition across embodiments, viewpoints, and scenes, as well as for video quality and downstream VLA adaptation. The experiments show that factorized recomposition generates useful robot-domain experience, including targeted interventions for state coverage and language-grounded spatial selection.
- Controllability and Video Quality: ActionAlign, CameraAlign, and EmbodAcc evaluate whether generated rollouts follow specified action, camera, and embodiment controls across 60 videos.The metrics use optical-flow similarity for action and camera alignment and CLIP-based binary accuracy for embodiment recognition.
- Qualitative Robot Experience Recomposition: AnyWorld recomposes human interactions across robot bodies, viewpoints, and scenes while preserving the underlying interaction structure.It can change the acting body, camera trajectory, background scene, or combinations of these factors while reusing action-camera controls.
- Controllability and Video Quality: The factorized model achieves the best action, camera, and embodiment controllability, while WAN Fun-Control lacks explicit camera modeling and Cosmos-Predict2.5 is weakly grounded in specified controls.The result supports the benefit of explicitly factorizing action, camera, and embodiment conditions.
- Controllability and Video Quality: The model remains competitive with or slightly improves over baselines on average VBench video quality without trading away controllability.Reported quality dimensions include subject consistency, background consistency, flicker stability, and motion smoothness.
- Training Ablations: The 2:1 human-to-robot fine-tuning mixture achieves the best average controllability by balancing human interaction diversity with robot-domain grounding.The 4:1 mixture has weaker CameraAlign and EmbodAcc, whereas the 1:1 mixture improves CameraAlign but lowers EmbodAcc relative to 2:1.
- Transferred Experience for VLA Adaptation: Transferred robot-domain rollouts improve VLA performance from 49.8% to 54.6% on 18 RoboCasa GR1 tasks and from 20.0% to 55.0% on 20 IRON banana-grasping trials.The generated data is mixed with target-domain robot data during adaptation at a 1:1 ratio.
- Targeted Capability Transfer through Controlled Recomposition: Controlled IRON interventions repair a spurious completion prior, while reliable spatial target switching requires calibrated actions grounded in re-embodied and re-scened visual pairs.Action-only counterfactuals do not reliably produce instruction following, whereas paired visual-action interventions do.
5 Conclusion
AnyWorld factorizes egocentric interactions into action, camera, and embodiment conditions to generate robot-domain rollouts without paired human-robot demonstrations. The model improves controllability, supports downstream VLA adaptation, and transfers targeted capabilities through robot-native visual recomposition.
- AnyWorld decomposes interactions into action, camera, and embodiment factors for controllable cross-embodiment rollout generation.It combines large-scale EgoDex pretraining with unpaired mixed-embodiment fine-tuning.
- The model supports recomposition across bodies, viewpoints, and scenes while preserving the underlying interaction structure.
- AnyWorld improves action-camera-embodiment controllability over strong video/world-model baselines while maintaining competitive video quality.
- Generated robot-domain experience improves UniT-based VLA adaptation in RoboCasa GR1 simulation and on a real IRON robot.
- Controlled capability interventions repair a spurious completion prior and transfer language-grounded spatial target selection.Action-only counterfactual pairing fails to explain the effect, indicating that robot-native visual recomposition is required to ground the transferred behavior.
6 Limitations
AnyWorld generates robot-domain visual experience, but its visual rollouts do not fully capture tactile feedback and contact forces needed for precise contact-rich control. The method also depends on reliable action-camera extraction and has currently been evaluated across only three embodiments.
- Visual rollouts cannot fully capture tactile feedback and contact forces required for precise contact-rich control.
- Severe occlusion, fast motion, tracking errors, or strong camera shake may reduce controllability by impairing action-camera extraction.
- The current evaluation covers 3 embodiments, leaving broader generalization to more robot morphologies, objects, and long-horizon tasks for future work.
Supplementary Material for AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization
The supplementary material accompanies AnyWorld with implementation details, analyses, and additional evaluations. The listed authors represent institutions including Nanyang Technological University, A*STAR, XPENG Robotics, Zhejiang University, and CUHK.
- The supplementary material is associated with Cheng Chen, Jerry Bai, Jiacheng Wei, Boyu Chen, and additional authors.
1 Overview of Supplementary Materials
The supplementary material provides implementation details, qualitative results, metric definitions, and downstream adaptation analyses for AnyWorld. It also describes reverse pseudo-pair construction, mixed-embodiment training, policy adaptation, and additional interaction-outcome evaluations.
- The supplement documents reverse pseudo-pair construction for human-to-robot embodiment editing and the mixed-embodiment training recipe.
- It includes structured conditioning, policy-adaptation details, and quantitative metric definitions.
- Additional evaluations analyze interaction outcomes under motion variations.
2 Implementation Details
AnyWorld’s embodiment editor converts human-like frames into target-robot images without paired human–robot supervision, preserving scene and interaction geometry. Reverse pseudo-pairs and curated training enable embodiment-specific editors for robot-native rollout initialization.
- Reverse pseudo-pair construction: Reverse pseudo-pairs convert robot frames into aligned human-like images, enabling supervised human-to-robot editing without real paired demonstrations.The strategy leverages an image editor’s stronger human-hand generation prior, then trains the inverse editing direction.
- Reverse pseudo-pair construction: Each target embodiment contributes approximately 300 high-quality pseudo pairs after removing edits with scene, camera, object, contact, geometry, or artifact errors.Approximately 800 robot frames are initially sampled per embodiment before manual curation.
- Embodiment editor fine-tuning: The embodiment editor takes a human-like source image and target-embodiment prompt, reconstructing the corresponding robot image with supervised flow matching.The diffusion transformer is fully fine-tuned while the pretrained tokenizer, image processor, text encoder, and VAE are retained.
- Embodiment editor fine-tuning: An embodiment-specific editor learns each robot’s morphology, material appearance, and spatial placement under the shared pseudo-paired formulation.At inference, the editor replaces visible human body parts while preserving task scene, object state, camera viewpoint, and target morphology.
- Qualitative embodiment editing: The resulting edited first frame supplies the target embodiment, scene, object state, and initial interaction geometry for AnyWorld rollouts.This image-editor bridge connects human egocentric demonstrations to robot-native world-model generation.
- Qualitative embodiment editing: Qualitative results show accurate replacement of human hands with IRON and RoboCasa GR1 hands while preserving location, contact geometry, scene context, and finger pose.The editor also handles a partially visible hand in the final example.
2.2 World Model Training Details
AnyWorld is trained in two full-fine-tuning stages: human egocentric pretraining establishes interaction priors, then mixed-embodiment data grounds them in robot domains. Generated re-embodied rollouts are added during policy adaptation before a shared real-robot fine-tuning stage.
- Backbone and fine-tuning strategy: AnyWorld uses full fine-tuning of the Wan 2.1 14B backbone and conditioning modules to jointly adapt motion, camera-following, and embodiment-conditioned appearance.The backbone is not adapted with parameter-efficient tuning.
- Human interaction pretraining: Human interaction pretraining on EgoDex uses aligned action, camera, first-frame, and text conditions to learn contact-rich egocentric interaction priors.This stage runs for 30K steps on 32 NVIDIA H200 GPUs.
- Mixed-embodiment fine-tuning: Mixed-embodiment fine-tuning combines unpaired EgoDex, IRON, and RoboCasa GR1 videos to ground pretrained interaction knowledge in target robot embodiments.This stage runs for 5K steps on 8 NVIDIA H200 GPUs.
- Structured embodiment conditioning: Structured captions separate scene, action, and subject descriptions so the same task and scene can be reused across embodiments by changing only the subject phrase.Cross-attention injects these captions, while the first frame specifies visual state and local geometry without an additional embodiment classifier.
- Policy adaptation: Generated re-embodied human rollouts are introduced during a 1K-step intermediate adaptation stage, preserving interaction structure, camera motion, and task semantics.The final 500-step stage uses only real target-robot data to reduce residual distribution mismatch.
- Targeted policy-gap interventions: Targeted IRON interventions jointly transfer visually re-embodied observations and morphology-calibrated wrist actions for policy-gap experiments.Action calibration includes resampling, interpolation, local deltas, PCA projection, scale calibration, and clipping.
- Targeted policy-gap interventions: Dual-target pairs keep the visual state fixed while reversing language instructions and aligned actions, supervising language-grounded target selection instead of fixed side bias.The final targeted set contains 26 left-target samples and 38 right-target samples.
2.5 Metric Details
AnyWorld is evaluated with controllability proxies for action, camera, and embodiment, plus controlled root-motion tests of interaction outcomes. The latter show that generated success or failure follows whether the imposed motion makes the target reachable.
- ActionAlign: ActionAlign measures local action-following by comparing background-compensated generated optical flow with control-skeleton flow inside the valid action region.Global background motion is estimated outside the action mask and subtracted before cosine similarity is computed.
- ActionAlign: Higher ActionAlign values indicate better consistency between generated motion and the input action-control video.The metric averages positive cosine similarity after background compensation.
- CameraAlign: CameraAlign compares source and generated background-motion vectors to assess whether the rollout follows the source camera trajectory.Median background flow approximates global pan, tilt, or translation, excluding action regions when masks are available.
- EmbodAcc: EmbodAcc is binary embodiment-recognition accuracy based on CLIP comparisons between cropped generated frames and human or robot prompt sets.Up to eight lower-region crops per video are evaluated because hands or robot arms are most likely to appear there.
- Aggregation: The reported metric means exclude invalid samples such as frames without reliable motion estimates or unavailable CLIP scores.Standard error of the mean is also computed when applicable.
- Physically consistent outcome simulation: Under fixed scene and embodiment conditions, varying root motion tests whether AnyWorld responds to embodied motion rather than merely producing plausible video.The evaluation keeps scene layout and object configuration fixed while changing root motion.
- Physically consistent outcome simulation: Valid root motion produces approach-and-grasp success, whereas spatially misaligned motion produces misses or incomplete grasps.These outcomes indicate dependence among global motion, object reachability, and downstream contact outcome.
- Physically consistent outcome simulation: The controlled outcomes provide positive and negative rollouts for downstream policy learning and evaluation.The model does not force success when the imposed motion is inconsistent with reaching the target.