HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit

Qingwei Ben; Feiyu Jia; Jia Zeng; Junting Dong; Dahua Lin; Jiangmiao Pang

Source-linked AI summary

HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit

Qingwei Ben, Feiyu Jia, Jia Zeng, Junting Dong, Dahua Lin, Jiangmiao Pang

arXiv:2502.13013v2Feb 18, 2025cs.ROcs.AIcs.HC

TL;DR

Generalizable humanoid loco-manipulation requires coordinated whole-body control and precise contact-rich manipulation, but existing systems fragment locomotion and teleoperation. HOMIE combines an RL body policy, isomorphic exoskeleton arms, motion-sensing gloves, and a pedal in one cockpit. The system supports stable walking and squatting with changing upper-body poses, faster task execution, and compatibility with multiple dexterous hands at low hardware cost.

Problem
Existing approaches lack a unified interface that combines adaptive locomotion with precise, full-body humanoid teleoperation.
Method
HOMIE combines a pedal-mapped RL locomotion policy, isomorphic exoskeleton arms, and motion-sensing gloves in a semi-autonomous teleoperation cockpit.
Results
HOMIE supports stable walking and squatting under dynamically changing upper-body poses, with faster task execution and gloves compatible with multiple dexterous hands.
Takeaways & Limitations
The cockpit enables a single operator to teleoperate humanoid whole-body movements for diverse loco-manipulation tasks in real and simulated environments.
Takeaways & Limitations
The reported scope centers on stable walking and squatting across diverse robotic platforms under dynamically changing upper-body poses.

Abstract

from arXiv · show

Generalizable humanoid loco-manipulation poses significant challenges, requiring coordinated whole-body control and precise, contact-rich object manipulation. To address this, this paper introduces HOMIE, a semi-autonomous teleoperation system that combines a reinforcement learning policy for body control mapped to a pedal, an isomorphic exoskeleton arm for arm control, and motion-sensing gloves for hand control, forming a unified cockpit to freely operate humanoids and establish a data flywheel. The policy incorporates novel designs, including an upper-body pose curriculum, a height-tracking reward, and symmetry utilization. These features enable the system to perform walking and squatting to specific heights while seamlessly adapting to arbitrary upper-body poses. The exoskeleton, by eliminating the reliance on inverse dynamics, delivers faster and more precise arm control. The gloves utilize Hall sensors instead of servos, allowing even compact devices to achieve 15 or more degrees of freedom and freely adapt to any model of dexterous hands. Compared to previous teleoperation systems, HOMIE stands out for its exceptional efficiency, completing tasks in half the time; its expanded working range, allowing users to freely reach high and low areas as well as interact with any objects; and its affordability, with a price of just $500. The system is fully open-source, demos and code can be found in our https://homietele.github.io/.

I. INTRODUCTION

HOMIE addresses the fragmentation between adaptive locomotion policies and precise teleoperation by unifying whole-body humanoid control in a single cockpit. Its RL training framework and low-cost hardware target stable loco-manipulation, efficient pose acquisition, and broader robot operation.

Motivation: Existing systems separate adaptive RL locomotion from precise teleoperation or omit locomotion, limiting either dexterous manipulation or workspace.RL policies lack real-time teleoperation interfaces, while upper-body systems do not account for locomotion’s effect on workspace.
System: HOMIE combines a pedal-mapped RL body policy, isomorphic exoskeleton arms, and motion-sensing gloves for single-operator full-body control.The cockpit integrates locomotion, arm, and hand control into one teleoperation system.
Training framework: The training framework uses an upper-body pose curriculum, height-tracking reward, and symmetry utilization to support walking, squatting, and dynamic upper-body movements.These techniques target balance adaptation, precise squatting, action regularization, and data augmentation.
Hardware and control: The pedal acquires locomotion commands while freeing the operator’s upper body, and the exoskeleton directly supplies upper-body joint angles.This enables simultaneous upper-body pose acquisition without continuous synchronized walking between operator and robot.
Hardware and control: $0.5k is the total hardware cost, with detachable gloves reusable across isomorphic robot systems.The system is presented as more affordable than motion-capture devices.
Evaluation: 200% faster and more accurate pose acquisition than previous methods was reported, while real-world studies showed stable diverse loco-manipulation in complex environments.Ablations evaluated the training techniques, and the cockpit also supported teleoperation in simulation.
Related systems: Prior approaches use costly identical robotic arms, VR or camera-based pose estimation, MoCap, or IK, while dexterous-hand support and accurate rapid control remain constrained.HOMIE is positioned against these cost, accuracy, speed, occlusion, IK, and dexterity limitations.

B. Whole-body Loco-Manipulation

HOMIE combines reinforcement-learning locomotion with exoskeleton-based upper-body teleoperation to support humanoid whole-body loco-manipulation. Its framework trains policies for walking and squatting under changing upper-body poses, with simulation training and zero-shot sim-to-real transfer.

RL-based methods provide an alternative to computationally demanding online optimal-control approaches for humanoid whole-body loco-manipulation.
HOMIE combines a low-level policy πloco with exoskeleton hardware for teleoperation in simulation and the real world.The system overview also includes demonstration collection for training an autonomous policy πauto.
The pedal supplies forward speed, turning speed, and torso-height commands, while the exoskeleton directly sets upper-body joint angles as πloco controls the lower body.Cockpit-robot communication uses Wi-Fi.
The RL framework trains different robots to walk and squat under continuously changing upper-body poses.The policy achieves zero-shot sim-to-real transfer, and the framework is based on prior locomotion implementations.
The policy observes commands, body state, joint positions and velocities, and the previous action across five consecutive steps, then outputs lower-body joint actions.The resulting actions are converted into motor torques through joint stiffness, damping, and default-position terms.

2) Upper-body Pose Curriculum:

The upper-body pose curriculum gradually broadens the training distribution while continuously changing target poses. This trains locomotion policies to handle dynamic upper-body movements across different robots.

2) Upper-body Pose Curriculum:: The curriculum starts with no upper-body action variation and increases the upper action ratio by 0.05 whenever velocity tracking reaches a reward threshold, until it reaches 1.
2) Upper-body Pose Curriculum:: Target upper-body poses are resampled every 1 second and uniformly interpolated so movements change gradually from the current pose to the desired pose.
2) Upper-body Pose Curriculum:: Different robots are trained in Isaac Gym to walk and squat while their upper-body poses continuously change.
2) Upper-body Pose Curriculum:: Tracking target heights expands the feasible operational workspace and supports additional loco-manipulation tasks.

3) Height Tracking Reward:

The height-tracking reward trains the robot to squat toward a target torso height while balancing squatting and walking during training. Symmetry augmentation and losses regularize left-right behavior.

3) Height Tracking Reward:: The knee reward encourages flexion when the robot height is below the target and extension when it is above the target.It uses the robot’s actual height, target height, and knee-joint action limits and positions.
3) Height Tracking Reward:: Commands are resampled every 4 seconds, assigning one-third of environments to squatting and two-thirds to standing and walking.The same environment alternates between squatting and walking, supporting smooth transitions.
3) Height Tracking Reward:: Symmetry augmentation flips actor and critic observations across the robot’s x-z plane, including left-right joint positions, velocities, actions, and turning velocity.
3) Height Tracking Reward:: Original and mirrored transitions are stored together to improve data efficiency and reduce asymmetry between left and right performance.
3) Height Tracking Reward:: The symmetry loss is defined as MSE(at, a′t) and is computed from actor and critic outputs for additional network-optimization losses.These losses enforce symmetry in the neural network.

C. Hardware System Design

HOMIE’s hardware cockpit combines isomorphic exoskeleton arms, motion-sensing gloves, and pedal-based locomotion control for single-operator humanoid teleoperation.

1) Isomorphic Exoskeleton:: The system uses symmetric 7-DoF isomorphic exoskeleton arms to map upper-limb joint angles onto Unitree G1 and Fourier GR-1.Each arm corresponds to three shoulder, one elbow, and three wrist degrees of freedom.
1) Isomorphic Exoskeleton:: Kinematic calibration uses an integer-multiple-of-π/2 offset, with direction and scale coefficient kt and compensation τt; the system sets kt = 1 and τt = 0.
2) Motion-sensing Gloves:: The gloves use Hall-effect sensors and neodymium magnets to capture finger joint angles and map dexterous-hand motions.Each finger has three sensor sets for fingertip and finger-pad pitch plus finger-pad yaw.
3) Foot Pedal:: The cockpit’s three pedals control yaw velocity, height, and linear velocity, while two buttons switch turning and travel directions.This interface leaves the operator’s upper body available for simultaneous pose acquisition.

3) Foot Pedal:

The pedal interface exposes locomotion commands directly, while ablations evaluate the training framework’s upper-body curriculum, height reward, and symmetry mechanisms.

3) Foot Pedal:: Three small pedals control yaw velocity, height, and linear velocity, with buttons switching left/right and forward/backward modes.High-precision rotary potentiometers convert pedal pressure into electrical command signals.
1) Ablation of training framework:: Ablations train Unitree G1 policies with three random seeds across 1000 environments for 20 seconds, changing only the relevant component.The experiments cover upper-body pose curriculum, height tracking reward, and symmetry utilization.
Upper-body Pose Curriculum.: The proposed upper-body curriculum achieves faster convergence and smaller linear-velocity, angular-velocity, and height-tracking errors than w/o cur and rand.The three settings show similar final symmetry loss and living time, while rand omits part of the intended difficulty range.
Height Tracking Reward.: The knee reward improves squat tracking: ours converges faster in height error than w/o knee and hei, while scaling the height reward alone harms multi-task balance.Linear and angular velocity errors remain similar for ours and w/o knee, and living time shows no significant difference.
Symmetry Utilization.: Symmetry data augmentation improves training efficiency, while symmetry loss reduces left-right symmetry breaking in the learned policy.The full method has slightly better overall tracking accuracy than augmentation alone.

2) Training on Different Robots:

The evaluation examines HOMIE across robot morphologies, hardware channels, dexterous hands, real-world tasks, and completion-time comparisons with OpenTelevision.

2) Training on Different Robots:: Fourier GR-1 is much taller and heavier than Unitree G1, yet the same training pipeline converges to robust walking and squatting policies under arbitrary upper-body poses.Only height-tracking ranges and robot-specific distance values change between the robot settings.
Hardware Evaluation: Joint matching produces high-frequency arm and hand outputs because robot poses are set directly without additional time-consuming processing.The approach does not require GPU- or SoC-intensive hardware for this output pathway.
Dexterous-Hand Extensibility: Motion-sensing gloves are tested with multiple dexterous hands in simulation, with figure rows identifying hand names and joint counts.The gloves are intended to support different dexterous-hand mappings.
Real-World Evaluation: Real-world Unitree G1 demonstrations include squatting to reach lower shelves, placing objects on higher shelves, and transferring boxes between shelves.The deployment uses 16 ms measured communication latency under normal network conditions.
Efficiency Comparison: Completion time is compared across Pick & Place, Scan Barcode, Hand Over, and Open Oven tasks between HOMIE hardware and OpenTelevision.The tasks exercise precise arm and hand control in varied desktop scenarios.

2) User Study:

User studies assess novice learning and simulation-based task deployment, with participants reaching expert-level performance after brief practice and policies migrating across simulated environments.

2) User Study:: Five novice testers with varied heights, weights, and genders progressively approached expert-level Hand Over completion times across five attempts.The study used identical starting positions and recorded tomato-transfer completion time after a brief tutorial.
2) User Study:: All testers achieved expert-level proficiency despite anthropometric differences from the system’s design specifications.Adjustable straps supported adaptability across diverse body types.
Simulation Migration: Policies trained for Unitree G1 and Fourier GR-1 migrate from Isaac Gym to GRUtopia-based simulated environments for broader loco-manipulation scenarios.The migration supports robot control across varied simulated scenes.
Demonstration Collection: The demonstration dataset contains RGB images, robot states, upper-body commands, and locomotion commands collected at 10 Hz, with 50 episodes per task.The Squat Pick and Pick & Place tasks are used to evaluate whether collected demonstrations support imitation learning.

2) Training Setting:

The training and evaluation pipeline combines visuomotor imitation learning with autonomous deployment to test complex whole-body loco-manipulation. Results indicate that collected teleoperation data can support stable real-world task execution, while the system remains limited by terrain reliability.

Training Setting: The visuomotor policy receives images and robot proprioceptive signals and continuously outputs control actions through an autoregressive transformer.Multi-view images use a MAE-pretrained ViT encoder, while proprioceptive features are extracted by an MLP and concatenated into tokens.
Evaluation: The trained autonomous policy is evaluated with task Success Rate after 15 trials per task.The policy is deployed on a humanoid robot, with results reported for the Squat Pick and Pick & Place tasks.
Results: Collected teleoperation data can drive robots to complete complex whole-body loco-manipulation tasks in the real world.The evaluation reports stable performance across diverse tasks and complex environments.
System: HOMIE combines a low-cost exoskeleton cockpit with an RL-trained loco-manipulation policy for whole-body humanoid control.The system supports teleoperation in simulation and the real world using an upper-body pose curriculum, height tracking, and symmetry-based training.
Limitations: The policies still do not reliably traverse diverse terrains, and the motion-sensing gloves can provide less intuitive operation for certain dexterous hands.The gloves also lack force feedback, and the exoskeleton does not provide waist teleoperation.

A. Network Architecture

The RL framework uses an estimator, follow-up, and critic network, together forming an actor-critic architecture for humanoid locomotion. Training modifies prior reward designs and applies domain randomization to improve real-world robustness and object handling.

A. Network Architecture: The network architecture contains estimator E, follow-up network N, and critic C, with E and N forming the actor module.All networks are implemented as 3-layer multilayer perceptrons.
A. Network Architecture: The follow-up network produces lower-body actions from estimator outputs and observations, while the critic receives observations and ground-truth velocity commands.The action dimension equals N_lower.
B. Reward Scales: The framework introduces a squat-knee reward and revises base-height tracking to follow changing target heights.Linear-velocity tracking is also decomposed into separate x- and y-velocity components.
B. Reward Scales: The reward scales used for Unitree G1 and Fourier GR-1 are largely similar, supporting the framework’s generality across heterogeneous robots.The paper reports this similarity as evidence of cross-robot generality.
Robustness: Domain randomization simulates deployment noise and randomizes hand mass to improve the robot’s ability to hold objects.The randomization ranges for Unitree G1 and Fourier GR-1 are the same.

D. Other Key Parameters

The system uses shared training parameters with robot-specific height targets and visualized reward functions, then extends the feasible workspace toward stair traversal. Hardware construction emphasizes low-cost, modifiable 3D-printed components and lightweight exoskeleton structures.

D. Other Key Parameters: Training and evaluation use the same key settings, with variable base-height targets for squatting and constant height tracking during walking.Values exceeding specified parameter ranges incur penalties through corresponding rewards.
D. Other Key Parameters: The proposed p(x|ra) distribution shifts toward larger x values as ra increases, becoming uniform on [0, 1] as ra approaches 1.The visualization explains how the curriculum parameter controls sampling behavior.
F. Terrain Traverse: The stair-traversal extension succeeds in simulation but remains unstable in real-world deployment.The robot collides with stairs, partly because head motion moves the LiDAR and the elevation map lacks high resolution.
Hardware: The hardware uses PLA 3D-printed structural components that are inexpensive, sufficiently strong, and easy to build or modify.The CAD models can also support construction with other materials.
Hardware: The exoskeleton uses hollowed-out mortise-and-tenon structures to reduce weight and assembly complexity while routing servo connections.Structural parts attach either directly to servo bodies or through servo disks.

B. Motion-sensing Glovesc Details

HOMIE’s hand and locomotion interfaces are designed for direct sensing across dexterous hands and foot-controlled movement commands. The cockpit combines gloves, pedals, and exoskeleton control in both teleoperation and autonomous deployment configurations.

B. Motion-sensing Gloves Details: The motion-sensing gloves track up to 15 DoF and directly map thumb, middle-finger, and index-finger movements to the Unitree Dex3-1 hand.The Dex3-1 has 7 DoF, while the gloves provide higher-dimensional sensing for direct correspondence.
C. Foot Pedal Details: Three pedals and two foot-operated mode buttons provide lower-body movement and directional commands.The pedal interface controls linear velocity, yaw velocity, and height adjustment through potentiometers and switching states.
C. Foot Pedal Details: The pedal’s spring-return mechanism converts foot pressure into potentiometer rotation and restores the pedal after release.The potentiometer has 270° angular movement, while the physical pedal movement is 40°.
Deployment: The teleoperation cockpit connects the isomorphic arm, gloves, and pedal to control the robot’s upper and lower body.The trained locomotion policy runs onboard at 50 Hz using robot state information for walking and squatting.
Deployment: The system supports both exoskeleton-controlled deployment and autonomous-policy deployment configurations.Autonomous deployment uses an RTX 4080 host and wired TCP communication to transmit image-based control information.

C. Simulation Deployment

HOMIE transfers policies between simulation platforms by aligning joint ordering and quaternion conventions, then evaluates user adaptability through a structured onboarding and task protocol. The protocol typically reaches task execution within 5 minutes, while testers with varied body types achieved expert-level proficiency after minimal practice.

Simulation Deployment: Isaac Gym policies are transferred to Isaac Sim by aligning joint order and quaternion conventions between the two platforms.Isaac Gym uses depth-first joint ordering and xyzw quaternions, whereas Isaac Sim uses breadth-first ordering and different conventions.
User Study: Testers spanning diverse heights, body types, and genders achieved expert-level handover proficiency after minimal practice with identical exoskeleton hardware.The result held despite anthropometric variation from the system’s design specifications.
Teaching Procedure: The onboarding protocol covers exoskeleton fitting, ergonomic calibration, pedal mapping, glove familiarization, arm coordination, and structured task execution.These phases pair operational objectives with verbal guidance examples.
Teaching Procedure: 5 minutes is the typical time from initial fitting to task execution under the standardized onboarding protocol.Task execution combines exoskeleton movements with pedal operations for loco-manipulation.