Source-linked AI summary
ExoActor: Exocentric Video Generation as Generalizable Interactive Humanoid Control
Yanghao Zhou, Jingyu Ma, Yibo Peng, Zhenguo Sun, Yu Bai, Börje F. Karlsson
TL;DR
Fluent interaction-rich humanoid behavior remains difficult to generalize across diverse environments and tasks. ExoActor uses third-person video generation as an intermediate representation, then converts generated videos into executable behaviors, demonstrating feasibility on real-world tasks without task-specific data collection.
Problem
Humanoid systems still struggle to generalize interaction-rich behaviors that jointly model spatial context, temporal dynamics, and task intent across diverse environments and tasks.
Method
ExoActor generates third-person task-execution videos from instructions and observations, then converts them into executable humanoid behaviors through motion estimation and control.
Results
Real-world experiments demonstrate the feasibility of using generated videos as actionable intermediate representations for humanoid control without task-specific data collection.
Takeaways & Limitations
ExoActor provides a scalable paradigm for modeling interaction dynamics with exocentric video generation while remaining compatible with existing control frameworks.
Takeaways & Limitations
The field lacks standardized benchmarks that evaluate video generation, motion estimation, and embodied execution jointly for video-driven humanoid control.
Abstract
from arXiv · showhide
Humanoid control systems have made significant progress in recent years, yet modeling fluent interaction-rich behavior between a robot, its surrounding environment, and task-relevant objects remains a fundamental challenge. This difficulty arises from the need to jointly capture spatial context, temporal dynamics, robot actions, and task intent at scale, which is a poor match to conventional supervision. We propose ExoActor, a novel framework that leverages the generalization capabilities of large-scale video generation models to address this problem. The key insight in ExoActor is to use third-person video generation as a unified interface for modeling interaction dynamics. Given a task instruction and scene context, ExoActor synthesizes plausible execution processes that implicitly encode coordinated interactions between robot, environment, and objects. Such video output is then transformed into executable humanoid behaviors through a pipeline that estimates human motion and executes it via a general motion controller, yielding a task-conditioned behavior sequence. To validate the proposed framework, we implement it as an end-to-end system and demonstrate its generalization to new scenarios without additional real-world data collection. Furthermore, we conclude by discussing limitations of the current implementation and outlining promising directions for future research, illustrating how ExoActor provides a scalable approach to modeling interaction-rich humanoid behaviors, potentially opening a new avenue for generative models to advance general-purpose humanoid intelligence.
1 INTRODUCTION
ExoActor addresses the challenge of modeling interaction-rich humanoid behavior by using third-person video generation as a scalable interface between task descriptions, scene context, and executable robot control. Its end-to-end system converts generated videos into motion-tracked behaviors without task-specific data collection, demonstrates feasibility across diverse tasks, and identifies remaining challenges for physically grounded generation and execution.
- Motivation: Humanoid robots still struggle to model fluent interactions among the robot, its environment, and task-relevant objects.These behaviors remain difficult despite advances in learning-based control and whole-body coordination.
- Approach: ExoActor uses third-person video generation to synthesize plausible task-execution videos from a task description and initial observation.The framework treats exocentric video generation as a scalable paradigm for modeling interaction dynamics while leveraging pretrained video models’ generalization.
- Approach: The system transforms generated videos into executable humanoid behaviors through whole-body and hand motion estimation followed by end-to-end motion execution.This decouples high-level interaction modeling from low-level control and remains compatible with existing control frameworks.
- Advantages: ExoActor eliminates task-specific data collection and provides a modular interface for long-horizon, interaction-rich behavior generation.Its video-generation, motion-estimation, and execution components can be improved independently.
- Validation and limitations: The implemented system demonstrates that generated videos can produce interaction-aware behaviors across diverse real-world humanoid tasks and difficulty levels.The paper also highlights future directions including physically grounded video generation, improved motion execution, vision-based whole-body control, and manipulation-intensive tasks.
2 METHOD
ExoActor converts high-level task instructions into executable humanoid behaviors through third-person video generation as an intermediate representation. Its pipeline aligns robot observations with human-centric video priors, structures task prompts, generates constrained action videos, and estimates interaction-aware human motion for robot execution.
- Overall Pipeline: ExoActor uses a three-stage pipeline that generates task-consistent third-person videos, estimates human motion, and converts the estimates into executable humanoid behaviors.The framework takes a task instruction and initial third-person observation as inputs and uses video as the intermediate representation.
- Robot-to-Human Embodiment Transfer: Robot-to-human embodiment transfer preserves scene layout, viewpoint, pose, orientation, scale, and body proportions while mapping robot observations into a human-like representation.This addresses unstable generation, distorted geometry, and degraded motion estimation caused by the mismatch between robot embodiments and human-centric model priors.
- Task-to-Action Decomposition and Prompt Construction: Each abstract instruction G is decomposed into a temporally ordered action chain C = {a1, a2, ..., aT} of atomic, observable, and physically executable actions.The decomposition preserves task intent, object attributes, and spatial relationships while removing redundant or ambiguous steps.
- Task- and Environment-Generalizable Video Generation: Structured prompt fields encode scene constraints, motion requirements, execution details, and end states to enforce fixed viewpoints, preserve geometry, and encourage plausible robot-aligned motion.The templates use the initial observation and enriched action description to reduce hallucinated or inconsistent behavior during video generation.
- Whole-body and Hand Motion Estimation: Interaction-aware motion estimation recovers 3D human trajectories from generated videos, while frame-wise hand estimation captures dexterous poses and maintains consistent correspondence with robot end-effectors.The recovered motion encodes object contacts, spatial constraints, task-relevant movements, grasp formation, release, and contact transitions for downstream execution.
3 EXPERIMENTS
Experiments evaluate ExoActor through zero-shot tasks spanning increasingly complex navigation, interaction, and manipulation demands. Case studies demonstrate behaviors across all difficulty levels, while ablations and failure analyses identify viewpoint, execution, and computational bottlenecks.
- Task hierarchy: Zero-shot tasks are organized into Level B (Easy), Level A (Moderate), and Level S (Challenging), reflecting increasing spatial reasoning, interaction complexity, and execution precision.The hierarchy enables structured evaluation of video-driven humanoid control.
- Case studies: At Level B, the robot reliably approaches target objects and navigates around obstacles, testing stable locomotion and spatially consistent goal reaching.Examples include navigating toward bottles or baskets and walking around chairs.
- Case studies: At Level A, the system executes multi-step behaviors combining navigation with coarse whole-body interactions, including sweeping, sitting, obstacle traversal, lifting, and wiping.These tasks do not require precise dexterous manipulation.
- Case studies: At Level S, the robot performs fine-grained multi-step manipulation, such as picking objects at different heights and placing or throwing them into target containers.Accurate locomotion and hand-object coordination are required, but residual hand-height errors may require supporting bases under target items.
- Limitations: Failure analyses identify hallucinated objects and inconsistent generated actions, inaccurate motion under occlusion or during wrist rotations, and execution mismatches in hand height and movement distance.These errors can propagate across video generation, motion estimation, and whole-body execution, especially when precise spatial alignment is required.
- Ablations and efficiency: Ablations find that direct motion input preserves geometric fidelity and positional accuracy, viewpoint preferences differ by task type, and video generation is the main runtime bottleneck.Exocentric back-to-front views favor navigation, front-facing views favor manipulation, and embodiment transfer adds relatively minor overhead.
4 DISCUSSION
The discussion identifies limitations in video realism, motion recovery, sensing assumptions, embodiment transfer, and open-loop execution, while proposing physically grounded, adaptive, and standardized alternatives. These directions aim to improve reliable transfer from generated interaction videos to executable humanoid behaviors.
- Embodied Control: Open-loop trajectory replay limits robustness to perception noise, localization errors, dynamic obstacles, and execution failures in changing environments.Closed-loop scene-aware control could treat generated motion as a high-level reference and adapt foot placement, posture, hand motion, and timing using visual and proprioceptive feedback.
- Video Generation Quality: Generated-video realism is a systemic bottleneck because spatiotemporal inconsistencies, incorrect contacts, and kinematic violations degrade motion recovery and real-world stability.Future models should incorporate object permanence, contact dynamics, geometric consistency, and motion constraints, while evaluation should measure physical plausibility and executability alongside visual fidelity.
- Observation and Deployment: The framework depends on external third-person views that may be unavailable or unreliable, motivating exocentric-video generation from first-person observations and onboard proprioception.This direction would preserve third-person interaction modeling while relaxing the external-camera requirement.
- Motion Estimation: Motion estimation is a major uncertainty source: modular pose, tracking, fitting, and retargeting stages accumulate errors, while wrist orientations remain ambiguous under occlusion and monocular views.Proposed remedies include jointly optimizing motion recovery and control, using embodiment-aware and task-oriented training, and adding geometric, kinematic, temporal, and object-aware constraints.
- Embodiment Generalization: Style transfer introduces a domain change that can weaken consistency between generated motions and robot execution, motivating robot-centric generation with smooth, continuous, multi-view-stable, physically coherent behavior.Such models should support robotic embodiments without merely rendering robots visually or inheriting rigid motion patterns from robot data.
- Evaluation: Standardized end-to-end benchmarks are needed because current evaluations separately assess video quality, motion reconstruction, and control performance.Unified evaluation should jointly measure video generation, motion estimation, and embodied execution.
5 RELATED WORK
Related work develops predictive world action models, learned simulators, and generalizable humanoid controllers, but still faces challenges in efficient long-horizon prediction and physically consistent deployment. ExoActor instead uses generative video as an intermediate interface to synthesize interaction dynamics and produce executable humanoid behaviors without an explicit world model or simulation loop.
- World action models: World action models predict future observations under agent actions to support control, policy learning, and decision making.Recent work increasingly emphasizes visually grounded predictive modeling for fine-grained, scene-dependent interaction dynamics.
- World action models: Current world action models struggle to jointly achieve long-horizon prediction, efficient inference, and physically precise anticipation.These limitations motivate more practical world action models for humanoid robotic control.
- Learned simulators: Learned simulators model environment evolution directly from data, offering flexible substrates for planning, optimization, and policy adaptation in visually rich settings.They complement classical simulators, which provide controllable and interpretable environments under simplified assumptions about geometry, contact, and material dynamics.
- ExoActor’s distinction: ExoActor uses generative video models to synthesize interaction dynamics and translate them into executable humanoid behaviors without an explicit world model or simulation loop.This differs from approaches centered on learned simulators for predictive modeling or policy training.
- Generalizable humanoid control: Generalizable humanoid control seeks robust whole-body behaviors across tasks, environments, and embodiments despite perception noise, motion variation, and distribution shift.Humanoids must coordinate locomotion, balance, upper-body motion, active perception, and task interaction within one control framework.
- Generalizable humanoid control: Existing studies advance semantic grounding, human-data transfer, and controller generalization, yet long-horizon compositionality, cross-embodiment robustness, and physically consistent deployment remain challenging.These unresolved issues limit reliable real-world humanoid control.
6 CONCLUSION
ExoActor uses exocentric video generation as an intermediate interface for modeling interaction-rich humanoid behavior and converting synthesized execution videos into executable actions. Real-world experiments demonstrate feasibility without task-specific data collection, while limitations remain in realism, translation, execution, and manipulation.
- Framework: ExoActor synthesizes third-person execution videos capturing coordinated humanoid, environmental, and object interactions before converting them into executable behaviors.The conversion pipeline uses motion estimation and general motion tracking.
- Empirical validation: Real-world experiments demonstrate that generated videos can serve as actionable intermediate representations for humanoid control without task-specific data collection.
- Broader significance: ExoActor connects generative video models with embodied humanoid control by enabling models to imagine, structure, and execute complex physical interactions.The work presents this as an initial step toward scalable humanoid systems.
- Limitations: Current limitations include physical realism, video-to-motion translation, open-loop execution, and manipulation-intensive tasks.
System Instruction
The system uses an action-decomposition prompt to convert high-level task goals into concise, temporally ordered robot-style action chains. These chains support third-person action video generation and robot motion retargeting.
- Action Decomposition: The action-decomposition prompt converts each high-level user task goal into a concise robot-style action chain paragraph.The prompt frames the model as a professional robotic action planner.
- Action Decomposition: The generated action chains are temporally ordered for third-person action video generation and robot motion retargeting.The ordering links task-goal decomposition with both video generation and downstream motion execution.
- Action Decomposition: Figure 11 presents the prompt used to decompose high-level task goals into robot-style action chain paragraphs.It documents the prompt design underlying the system instruction.
A APPENDIX · A.1 ACTION DECOMPOSITION ORGANIZATION
The appendix describes an action-decomposition prompt that converts high-level task goals into temporally ordered robot-style action chains for grounded third-person video generation. It specifies intermediate motions and interaction details to support physically plausible synthesis and downstream motion execution.
- A APPENDIX: The prompt converts high-level task goals into temporally ordered robot-style action chain paragraphs.It provides the organizational structure described for action decomposition.
- A.1 ACTION DECOMPOSITION ORGANIZATION: The action chains explicitly specify intermediate motions within each task.These intermediate motions make the intended execution process more concrete.
- A.1 ACTION DECOMPOSITION ORGANIZATION: The prompt also specifies interaction details between the robot and task-relevant elements.Interaction details are included alongside the ordered motions.
- A.1 ACTION DECOMPOSITION ORGANIZATION: This structured representation provides a consistent interface for third-person video generation.The interface is designed around the organized action-chain representation.
- A.1 ACTION DECOMPOSITION ORGANIZATION: The interface is visually grounded so synthesized actions remain physically plausible.Visual grounding is presented as a condition supporting plausible generated actions.
- A.1 ACTION DECOMPOSITION ORGANIZATION: The decomposition keeps generated actions aligned with downstream motion extraction and execution.The prompt is intended to connect video synthesis with later behavior realization.
A.2 EMBODIMENT TRANSFER PROMPT
The embodiment-transfer prompt enforces one-to-one robot–human alignment while preserving the original scene and camera, using structured initial-state information extracted from the first frame.
- The system prompt in Fig. 12 transfers robot embodiment to a human subject for execution.
- Strict pose, orientation, and scale constraints enforce one-to-one alignment between the robot and generated human subject.
- The prompt preserves the original scene and camera configuration during embodiment transfer.
- Gemini 3.1 produces structured initial-state information extracted from the first frame according to the prompt.
A.3 ACTION-CHAIN-CONDITIONED GENERATION PROMPT CONSTRUCTION
The method converts a decomposed action chain, initial observation, and task goal into a scene- and task-aware prompt for third-person video generation. It preserves temporal structure while grounding generation in spatial details and fixed prompt constraints.
- Prompt construction: The final video-generation prompt grounds the planned action sequence in the initial observation and original task goal.This construction is illustrated in Figure 13.
- Prompt construction: A multimodal prompt engineer combines visual observation, action chain, and action goal into a scene- and task-aware action description.The description is intended for third-person action video generation.
- Prompt construction: The action chain supplies temporal structure, while initial-state information specifies pose, orientation, object locations, and spatial relations.These inputs are combined into a spatial- and task-aware description before insertion into a fixed generation prompt template.
- Prompt construction: The fixed generation prompt template further constrains camera viewpoint, scene consistency, and motion stability.The passage also indicates constraints related to execution, but its final wording is truncated.
A.4 B-LEVEL NAVIGATION TASK GENERATION PROMPT · A.5 A/S-LEVEL ADVANCED TASK GENERATION PROMPT
The appendix defines fixed-camera prompt templates for B-level navigation and A/S-level advanced task generation. Both target stable, temporally coherent, physically plausible human-action videos suitable for downstream motion extraction and execution, while advanced tasks add manipulation and long-horizon constraints.
- A.4 B-LEVEL NAVIGATION TASK GENERATION PROMPT: B-level navigation prompts enforce fixed-camera, scene-consistent settings and natural, physically plausible navigation behaviors.These constraints are intended to keep generated videos stable and suitable for downstream motion extraction and execution.
- A.5 A/S-LEVEL ADVANCED TASK GENERATION PROMPT: A/S-level advanced prompts add manipulation and long-horizon constraints, explicitly specifying task structure, interaction details, and robot-like motion characteristics.The formulation encourages physically plausible, temporally coherent behaviors suitable for downstream motion extraction and execution.
- A.4 B-LEVEL NAVIGATION TASK GENERATION PROMPT: B-level generation specifies a task-consistent human-action video under a fixed third-person camera view.The template describes a real indoor scene captured in a locked-off 10-second shot.
- A.4 B-LEVEL NAVIGATION TASK GENERATION PROMPT: The B-level shot keeps the camera completely fixed, without movement, zoom, reframing, or perspective change.The prescribed duration is 10 seconds.
- A.4 B-LEVEL NAVIGATION TASK GENERATION PROMPT: Figure 14 presents the prompt template used for B-level navigation task generation.The figure documents the implementation template described in the appendix.
- A.5 A/S-LEVEL ADVANCED TASK GENERATION PROMPT: A/S-level generation requests task-consistent human-action video for advanced manipulation or long-horizon tasks under a fixed third-person camera view.Its shot specification uses a 10-second locked-off indoor scene with fixed viewpoint, framing, focal length, zoom, and perspective.
- A.5 A/S-LEVEL ADVANCED TASK GENERATION PROMPT: The advanced shot prohibits camera movement or reframing while preserving fixed viewpoint, framing, focal length, zoom, and perspective.The specification describes a real indoor scene in a 10-second locked-off static shot.
- A.5 A/S-LEVEL ADVANCED TASK GENERATION PROMPT: Figure 15 presents the prompt template used for A/S-level advanced task generation.The figure documents the advanced-task implementation template described in the appendix.