Source-linked AI summary
EgoActor: Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-Language Models
Yu Bai, MingMing Yu, Chaojie Li, Ziyi Bai, Xinlong Wang, Börje F. Karlsson
TL;DR
Humanoid robots must coordinate perception, locomotion, manipulation, and interaction under partial information and changing environments. EgoActor addresses this through EgoActing, a VLM that predicts spatially grounded humanoid actions from egocentric observations, with evaluations showing generalization across diverse tasks and unseen environments. Its effectiveness remains dependent on external planners and downstream control or manipulation components.
Problem
Real-world humanoid deployment requires coordinated perception, locomotion, manipulation, and transitions between action types under partial-information, dynamic conditions.
Method
EgoActor is a unified VLM that transforms instructions and egocentric observations into locomotion, head movement, manipulation, and human-interaction actions.
Results
EgoActor demonstrates strong generalization and timely inference in extensive simulation and physical-robot evaluations across diverse tasks and unseen environments.
Takeaways & Limitations
EgoActor bridges abstract task planning and low-level humanoid action execution using RGB-only observations and scalable heterogeneous training data.
Takeaways & Limitations
EgoActor depends on external high-level planners and downstream locomotion and manipulation models, and can make locally optimal errors in long multi-stage tasks.
Abstract
from arXiv · showhide
Deploying humanoid robots in real-world settings is fundamentally challenging, as it demands tight integration of perception, locomotion, and manipulation under partial-information observations and dynamically changing environments. As well as transitioning robustly between sub-tasks of different types. Towards addressing these challenges, we propose a novel task - EgoActing, which requires directly grounding high-level instructions into various, precise, spatially aware humanoid actions. We further instantiate this task by introducing EgoActor, a unified and scalable vision-language model (VLM) that can predict locomotion primitives (e.g., walk, turn, move sideways, change height), head movements, manipulation commands, and human-robot interactions to coordinate perception and execution in real-time. We leverage broad supervision over egocentric RGB-only data from real-world demonstrations, spatial reasoning question-answering, and simulated environment demonstrations, enabling EgoActor to make robust, context-aware decisions and perform fluent action inference (under 1s) with both 8B and 4B parameter models. Extensive evaluations in both simulated and real-world environments demonstrate that EgoActor effectively bridges abstract task planning and concrete motor execution, while generalizing across diverse tasks and unseen environments.
I. INTRODUCTION
EgoActing targets the challenge of transforming high-level instructions into spatially grounded, executable humanoid action sequences. EgoActor unifies movement, perception, manipulation, and human interaction using broad supervision and validates this approach in real-world and simulated settings.
- Real-world humanoid deployment remains difficult because instability, partial-information tasks, timing, precision, and obstacle handling can produce unsafe behavior.
- EgoActing requires transforming actionable instructions into situated action sequences using egocentric observations, action history, and available skills.
- EgoActor predicts locomotion, active perception, manipulation, and human-interaction actions to ground high-level instructions into executable humanoid behaviors.Its movement outputs include forward motion, strafing, turning, and postural adjustments; active perception includes head orientation actions.
- EgoActor is trained on diverse real-world video, spatial reasoning, action-timing, and virtual-environment data, supporting 8B and 4B models with sub-second inference.
- The framework is evaluated through extensive real-world and simulated experiments, with open-source code, models, datasets, and evaluation protocols released for reproducibility.
- Unlike approaches centered on predefined skill libraries, EgoActor directly predicts egocentric low-level actions such as locomotion and head movement.
B. Mobile-Manipulation
EgoActing extends mobile-manipulation beyond navigation by grounding instructions into coordinated, egocentric humanoid actions. The task combines observations, action history, explicit instructions, and available control policies across movement, perception, manipulation, and interaction.
- EgoActor jointly reasons over locomotion, posture, spatial perception, manipulation, and human interaction within a single VLM-based action predictor.
- EgoActor addresses a broader scope than conventional VLN and object-goal navigation by coordinating whole-body behaviors for dynamic real-world humanoid control.
- The task predicts the next concrete humanoid actions from an actionable instruction, egocentric observation history, past actions, and available low-level policies.
- Instructions specify intended movements and goals, including spatial and temporal guidance, without prescribing low-level motor details.
- EgoActing includes active perception, manipulation, human-interaction, and movement as candidate action categories.
- Lateral movement and postural adjustments support obstacle avoidance, target alignment, and manipulation across varying heights and spatial constraints.
B. Language-based Actions
EgoActor represents robot behaviors as textual actions, combining structured language for precise spatial positioning with natural language for open-ended manipulation and interaction. This representation links perception-driven movement to subsequent task-specific actions.
- Structured language actions describe movement and active perception using interpretable action type, direction, and magnitude templates.Examples include turning left by a specified angle and looking up by a specified angle.
- Structured actions cover rotations, forward–backward and lateral translations, and vertical adjustments, while thresholds filter negligible movements.Their purpose is to interpret spatial relationships from RGB observations and position the robot for subsequent actions.
- Natural language actions represent manipulation and human-interaction behaviors without restricting the system to a fixed skill set.
- Natural language actions support generalization beyond predefined primitives and can express previously unseen actions.
- In the working procedure, grey SLA blocks encode structured movement or perception actions, while green NLA blocks encode natural-language actions.
- The model can generate context-specific interaction commands, such as asking a person to guide it to a meeting room.
IV. TRAINING RECIPE
EgoActor adopts a general VLM architecture and trains it with heterogeneous egocentric, spatial, planning, and simulated data. The recipe combines scalable supervision, temporal context, and manually or automatically derived action labels.
- The model uses a general VLM architecture to avoid specialized modifications and benefit from scalable video training data.
- Training uses randomly mixed data from all sources for both 4B and 8B model variants.
- Each prediction uses recent observation-action pairs together with sampled historical observations to provide temporal context.
- Demonstrations from virtual and real environments receive concise textual trajectory descriptions that identify target objects and movement routes.
- The training mixture includes internet and local egocentric videos, virtual navigation and EgoActing trajectories, spatial reasoning, visual-language understanding, and planning data.
- Unsupervised movement-transition data and DAgger experience data add spatial grounding and real-world execution trajectories.
D. Skill Setup
The skill setup combines dedicated manipulation and locomotion controllers with calibrated humanoid motion primitives, RGB-only sensing, and navigation-model baselines. EgoActor inference uses stochastic sampling, while baselines use their original greedy-decoding settings.
- Controllers: Manipulation uses a finetuned GROOT-N 1.5 model, while locomotion adopts Unitree’s official walking policy.
- Controllers: Motion calibration targets approximately 5 cm positional precision and about 5 degrees turning precision.
- Motion processing: Forward and turning actions are merged, and predicted forward distance is amplified by 1.2× for faster, more human-like movement.
- Inference: EgoActor uses stochastic sampling at temperature 0.2, whereas baseline models follow their original settings with greedy decoding.
- Robot setup: Real-world experiments use a Unitree G14 with Dex3-1 hands, a custom 2-DoF head, and a RealSense D455 capturing 480p monocular RGB without depth.
- Baselines: Navigation baselines include NaVid, Uni-NaVid, and NaVILA, representing video-based, unified, and two-level spatially grounded approaches.
C. Real-world Benchmarking
Real-world benchmarks test human interaction, mobile manipulation, and traversability across people, objects, unseen layouts, and narrow passages. EgoActor generally supports these combined behaviors, with stronger performance from the 8B model in demanding person-identification cases.
- Human-robot interaction: Human-interaction trials require navigating to a specified person and performing actions such as greeting, requesting information, or asking for help.
- Human-robot interaction: Experiments use people with unseen appearances and clothing, while person arrangements are randomized to test generalization and disambiguation.
- Human-robot interaction: Both EgoActor sizes generally approach people and perform basic interactions, but the 4B model is weaker with multiple people and fine-grained attributes.
- Mobile manipulation: Under an unseen layout, the 8B model generally reaches correct objects and supports manipulation for both in-distribution and out-of-distribution categories.
- Traversability: Traversability focuses on safely navigating narrow everyday spaces without collisions, a setting where current VLM navigation models often hit obstacles.
- Traversability: The real-world evaluation covers five rooms, including three seen meeting rooms and two unseen rooms, with entry and exit tested from two starting positions.
- Traversability: EgoActor generally traverses narrow passages and avoids collisions more reliably than baseline VLM navigators, with qualitative robustness on unseen layouts and obstacles.
D. Virtual Environment Benchmarking
Virtual benchmarking evaluates EgoActor on labeled samples from environments unseen during training, alongside qualitative traces of traversability, height changes, obstacle avoidance, active perception, and spatially adaptive movement. Results show comparable 4B and 8B performance overall, with failures concentrated in ambiguous or visually degraded settings.
- Benchmark setup: The virtual evaluation uses 205 labeled EgoActing samples from environments unseen during training and averages three stochastic test runs.
- Qualitative traces: The traversability trace shows the robot walking through a doorway, while the height-change trace demonstrates posture adjustment in virtual environments.
- Quantitative results: EgoActor generalizes well to unseen environments and target objects, with 8B slightly better at smaller distance thresholds and comparable overall 4B and 8B performance.
- Failure analysis: Most virtual-environment errors arise from ambiguous instructions or blurry environments, with additional failures in unfamiliar scene types.
- Baseline comparison: Under the standard VLN criterion of <3.0 m, baseline performance remains around 50%, but stricter interaction-positioning criteria frequently expose stopping failures.
- Active perception: The model looks downward to verify obstacles and fixes its gaze on targets to support smoother transitions into manipulation or interaction.
- Spatial and human-like behavior: Human-video training is associated with spatially adaptive distances and turning angles, including backward movement, strafing, and combined turning behaviors.
VI. CONCLUSION
EgoActor is a unified vision–language model for grounding high-level intentions into egocentric, executable humanoid actions. Its structured prompting uses visual context, temporal history, and constrained low-level action outputs.
- EgoActor jointly predicts locomotion, manipulation, human interaction, and head movements without requiring extra sensing modalities, multiple cameras, or extensive teleoperation.
- The model depends on external high-level planners and downstream locomotion and manipulation skills, so it is not fully end-to-end.
- EgoActor may select locally optimal but incorrect decisions during extended or multi-stage tasks because of limited long-term context.
- The EgoActing prompt exposes egocentric visual context, temporal history, and recent action–observation pairs for first-person reasoning.
- The output format constrains predictions to executable actions including turning, looking, moving, manipulation, interaction, and stopping.
A. EgoTaskQA Data Processing
EgoTaskQA sequences are converted into structured EgoActing samples that align instructions, historical observations, recent action–observation pairs, and temporally grounded supervision. Navigation and manipulation actions are normalized into executable training targets.
- EgoTaskQA supplies fine-grained temporal annotations that support constructing sequential decision-making data from event-understanding sequences.
- Each processed sample contains a global instruction, historical observations, and three interleaved recent observation–action pairs.
- Three adjacent annotated actions are concatenated into one instruction, while the start frame is placed 60 frames before the first action when possible.
- Recent actions represent manipulation starts, aggregated movement segments, or a Stop and no action token after the final manipulation.
- Navigation actions aggregate camera pose changes and retain motions exceeding 5 degrees of angular movement or 0.1 meters of translation.
- The pipeline produces instruction-conditioned samples by aligning RGB observations with normalized, merged actions in sliding windows.
X. SUPPORTED SKILLS
EgoActor supports movement, perception, manipulation, and human–robot interaction through structured spatial actions and flexible natural-language commands. Its evaluation includes traversability across seen and unseen real-world rooms.
- Supported skills span movement, perception, manipulation, and human–robot interaction, represented through structured and natural-language actions.
- Structured actions specify action type, direction, and magnitude for locomotion, body-height adjustment, lateral motion, and head orientation.
- Natural-language actions cover object manipulation, communication, and gestures, enabling flexible composition and generalization to unseen actions.
- Traversability evaluates safe room entry and exit through narrow doorways, including three training-seen rooms and two unseen environments.
- Each room is tested for entering and exiting from two doorway-relative starting positions with four repeated trials per position.
XIV. ADDITIONAL CASE STUDY
Additional cases illustrate sub-second action prediction, adaptive mobile manipulation, obstacle avoidance, and natural-language human interaction. The examples span cluttered, narrow, unseen, and virtual environments.
- EgoActor’s action prediction operates sub-second, while supplemental-video latency is mainly attributed to network transmission and streaming conditions.
- During mobile manipulation, the robot uses larger approach steps, then smaller movements to align precisely with a previously unseen pink cup.
- The pink cup is successfully picked up despite a pen holder on the desk, demonstrating spatial adaptation under clutter.
- Obstacle-avoidance examples show locomotion primitives adapting to static and dynamic obstacles in unseen narrow layouts while preserving clearance and task progress.
- Human-interaction examples include context-appropriate requests and inquiries despite fewer than 20 training samples for the task.
- EgoActing examples pair high-level goals with concrete natural-language actions across navigation, object interaction, and human interaction in real and virtual environments.
XVII. DIFFERENCE BETWEEN OUR WORK AND EXISTING
EgoActor differs from existing embodied systems by targeting full humanoid embodiment, unifying heterogeneous action reasoning, and extending beyond navigation to dynamic, long-horizon tasks in unseen environments.
- a) Scope of embodiment.: EgoActor targets full humanoid robots and directly predicts locomotion, posture, head orientation, manipulation, and human-interaction actions.This contrasts with prior VLM-based agents and LLM-driven systems focused on simplified embodiments or predefined skill libraries.
- b) Unified action reasoning.: EgoActing and EgoActor jointly reason over heterogeneous action types to generate temporally coherent, context-aware sequences without explicit intermediate planning.They unify perception, locomotion, and manipulation rather than decomposing tasks into modular subgoals or stagewise controllers.
- c) Task generalization and real-world deployment.: The approach extends beyond static navigation and object localization to dynamic, long-horizon tasks combining movement, manipulation, active perception, and human interaction.The supplied examples include instruction–action pairs, human-interaction prompts, mobile-manipulation tasks, and room navigation and interaction prompts.