Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand

Amirhossein Kazemipour; Hehui Zheng; Robert Katzschmann

Source-linked AI summary

Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand

Amirhossein Kazemipour, Hehui Zheng, Robert Katzschmann

arXiv:2609.17172v1Sep 15, 2026cs.ROeess.SY

TL;DR

A robotic hand must move, support itself, and manipulate objects with the same unequal fingers, without a separate locomotion mechanism. The paper trains stance-aware task policies in a hardware-calibrated simulator while retaining the hand’s design and position controller. The resulting system moves faster than tuned quadruped rewards in simulation and demonstrates untethered locomotion, recovery, keyboard pressing, and vision-guided pushing on hardware.

Problem
A mobile robotic hand needs to coordinate locomotion, self-support, and manipulation with the same unequal fingers while retaining its existing design and controller.
Method
The paper uses a stance-based footprint reward, task-specific PPO policies, and a simulator calibrated to hardware measurements while preserving the hand’s finger kinematics and position controller.
Results
The hand moves faster with the proposed reward than with tuned quadruped rewards in simulation and demonstrates untethered crawling, recovery, keyboard pressing, and vision-guided object pushing on hardware.
Takeaways & Limitations
A commercial anthropomorphic hand can serve as a self-contained mobile manipulator using its fingers for locomotion, support, and environmental interaction.

Abstract

from arXiv · show

A walking robotic hand must use the same fingers to move its body, support its weight, and interact with the environment. We show how an anthropomorphic hand can learn these skills while retaining its finger design and position controller. Onboard power and computation make the platform self-contained. Our reinforcement learning approach accounts for the hand's unequal fingers, with training in a simulator calibrated from hardware measurements. In simulation, the hand moves faster with our reward formulation than with tuned rewards originally designed for quadrupeds. On hardware, task-specific policies enable untethered crawling, steering, and fall recovery. While supporting its own weight, the hand also executes successive keyboard commands without vision and pushes an object to targets using overhead visual feedback. These results demonstrate a compact mobile manipulator that reuses its fingers for locomotion and interaction, without a separate locomotion mechanism.

I. INTRODUCTION

The paper turns an off-the-shelf anthropomorphic hand into a self-contained mobile manipulator by reusing its unequal fingers for locomotion, support, and interaction. A stance-based reward and calibrated simulation enable untethered task-specific behaviors, including crawling, recovery, keyboard pressing, and object pushing.

Challenge: Unequal fingers, an opposed thumb, and a tilted palm make balance and commanded motion difficult when contacts are removed for stepping or interaction.Remaining fingertips must support the hand while another finger steps or presses a key.
System: An 818 g commercial hand retains its finger kinematics and position controller while carrying its own battery and computer for untethered locomotion and interaction.The same fingers walk, support the hand, and interact with the environment.
Approach: A stance-based footprint reward pulls each fingertip toward its own nominal position while leaving the policy to learn where and when fingers step.Commands use a control frame that removes the nominal palm tilt, and each task is trained in a simulator calibrated to hardware measurements.
Evaluation: The platform demonstrates untethered crawling, steering, fall recovery, vision-free successive keyboard pressing, and overhead-vision object pushing.These task-specific policies reuse the fingers for both self-support and environmental interaction.

II. RELATED WORK

Prior work explores mobile fingers, leg-assisted manipulation, morphology-aware learning, gait rewards, and sim-to-real transfer. This paper differs by using an asymmetric commercial hand whose fingers must simultaneously provide locomotion, support, and manipulation.

Manipulation with legs: The hand must balance interaction and support because its fingers provide all body support, unlike systems with separate legs or locomotion mechanisms.A finger used to press or move an object is temporarily unavailable for support.
Morphology and reward design: An opposed thumb and four unequal fingers lack the left–right symmetry required by common mirroring methods for locomotion learning.The morphology therefore motivates a reward and policy formulation that does not rely on mirrored states and actions.
Morphology and reward design: The footprint objective anchors each fingertip to a nominal place in a level stance frame, while stepping timing and distance remain policy decisions.This contrasts with gait rewards that prescribe contact schedules, phases, or fixed foot-position timing.
Sim-to-real transfer: Hardware measurements inform actuator and contact models so policies can transfer through the hand’s retained position-control interface.The onboard module provides power, sensing, and computation for the self-contained platform.

B. Policy interfaces

Each task policy maps proprioceptive histories and task inputs to incremental joint-target commands, which the hand’s built-in position controller tracks. The interface preserves the same observation and action transformations for onboard deployment.

Policy interface: Each feedforward PPO policy runs at 50 Hz and maps short proprioceptive histories plus task inputs to joint-position increments.The retained position controller then tracks the resulting targets onboard.
Common observations: All policies share 46 inputs covering relative joint angles, gravity direction, angular velocity, and the previous policy output.Inputs are expressed partly in the palm frame and retain eight samples before normalization.
Task inputs: Task-specific information is supplied in stance-calibrated frame V, with positions in metres and velocities in metres per second.The table organizes the additional actor inputs beyond the common observation terms.
Action interface: At each step, the controller adds a scaled tanh-transformed policy output to the previous joint target and enforces joint limits.The angle scale differs by task, while simulation and deployment use the same observation and action transforms.

C. Hardware-calibrated simulation

The simulator is calibrated with measured actuator and contact behavior, then used to train PPO policies for commanded crawling with hardware-relevant disturbances and termination conditions.

Hardware-calibrated simulation: Measured joint stiffness, fingertip friction, delay, filtered joint speed, and filter cutoff parameterize the simulator’s position-controlled hand model.These quantities come from frequency sweeps, loaded fingertip pulls, and timed command responses.
Learning problem: The crawl policy follows planar velocity and yaw-rate commands while supporting the hand on its fingertips, with episodes ending after specified body-contact, tilt, or timeout events.Command ranges are vx ∈[0, 0.16] m/s, vy ∈[−0.16, 0.16] m/s, and ωz ∈[−0.45, 0.45] rad/s.
Optimization: Separate PPO actor and critic networks are trained in NVIDIA Isaac Lab, with the actor receiving observations available on hardware and the critic receiving privileged simulator state.Physics runs at 200 Hz, with one policy action every four physics steps.
Domain randomization: Training randomizes friction, effort scale, payload center-of-mass offsets, palm properties, IMU bias, and actuator gains around calibrated values.The deployed crawl policy therefore includes variation in both contact conditions and hardware dynamics.

B. Stance-calibrated fingertip objectives

The method calibrates fingertip objectives to the hand’s unequal stance geometry and nominal palm tilt, while letting the policy learn stepping timing. A footprint penalty anchors each fingertip spatially, and auxiliary objectives shape lifting, direction, speed, and turning.

Stance-calibrated control frame: Each unequal fingertip is assigned a nominal stance target in a level, stance-aligned frame that follows body rotation and removes nominal palm tilt.The frame is calibrated from a settled 80 g-payload stance; commands and base-relative fingertip kinematics are expressed in it.
Footprint objective: The footprint objective penalizes lateral and vertical fingertip departures four times more strongly than fore–aft deviations, leaving room for stepping.Its virtual-spring interpretation uses softer fore–aft and stiffer lateral and vertical reward weights.
Learned stepping: The targets move with the body and do not prescribe ground locations or a footfall sequence, so the policy learns where and when each finger steps.Lift and direction objectives provide additional frequency and swing-direction shaping during training.
Auxiliary lift objective: A command-dependent lift objective shapes stepping frequency through phase, airborne-contact gating, smoothstep velocity scaling, and a scheduled lift-weight decay.The frequency mapping spans 1 to 3.5 Hz as commanded speed increases from 0.02 to 0.16 m/s, with endpoint clamping.
Full reward formulation: The full reward tracks commanded planar velocity and delayed yaw rate while penalizing undesired contacts, body motion, joint effort, acceleration, and action changes.Yaw tracking is introduced after 18,000 steps so the policy does not learn stepping and turning simultaneously.

C. Fall recovery

A separate recovery policy enables the hand to regain its crawl stance after falling, without human repositioning. It learns upright stabilization from randomized side-lying starts and then transitions the hand to rest.

Fall recovery: Fall recovery starts from randomized side-lying poses and rewards level palm orientation, crawl-stance height, and joint pose without early termination.Each episode lasts 20 s, and the same PPO setup is used for the recovery policy.
Fall recovery: After becoming upright, the policy maintains balance through continuous joint motion before a smooth joint-target ramp brings the hand to rest.

V. SELF-SUPPORTED MANIPULATION POLICIES

The manipulation policies reuse calibrated simulation, proprioceptive support requirements, and PPO for keyboard pressing and visually guided object pushing. They combine self-support with control interaction, using known keyboard alignment or delayed noisy overhead vision as appropriate.

Policy design: Separate keyboard and object-pushing policies retain the calibrated simulator, proprioceptive history, support requirements, and PPO implementation.Both policies use normalized observations, ELU activations, and privileged simulation state for the critic during training.
Keyboard pressing: Keyboard operation uses a one-hot key command to select four learned press locations, with trial presses aligning the keyboard before each evaluation block.Successive commands depend on maintained alignment, and misalignment requires manual realignment.
Object pushing: Overhead vision tracks a dorsal marker and object, while training models 60 Hz sample-and-hold sensing, one-to-three-step latency, 3% dropout, and 2 mm position noise.The target remains fixed after each trial begins, and the policy learns approach, contact, and pushing jointly.
Object pushing: The pushing reward combines object approach, motion toward the target, target retention, fingertip support, and regularization.
Evaluation scope: The experiments first test the reward formulation in simulation, then evaluate untethered movement, stance recovery, and environmental interaction.

A. Reward-term ablation in simulation

The ablation compares the hand-specific reward with quadruped-derived rewards and tests how footprint, lift, and direction terms affect speed, participation, and contact posture. The footprint objective provides the clearest supported gains, while weight changes expose trade-offs between participation and posture.

c) Comparison with stock rewards: 0.65 cm/s faster on average than tuned stock, our formulation wins in 10 of 12 seeds.The 95% confidence interval for the difference is [+0.26, +1.02] across twelve seeds.
c) Comparison with stock rewards: Our formulation has higher non-nail participation by +0.35 [+0.18, +0.53], although tuned stock’s higher mean five-finger participation is uncertain.Tuned stock plants fingers on their nails more in every seed, so participation alone does not capture contact posture.
d) Contribution of individual objectives: Removing the footprint objective reduces task speed by 0.79 cm/s [+0.37, +1.19] and lowers five-finger participation.At the deployed weight, it shifts mean contact tilt across fingertips 8.8° toward the nail side ([+1.4, +15.9]).
d) Contribution of individual objectives: The ablations do not establish an independent speed benefit from the auxiliary lift or direction objectives.Removing direction alone increases five-finger participation by 0.31 [+0.07, +0.52], but its contact-posture effect remains uncertain.
e) Contact posture and reward weights: Doubling footprint or lift weight raises five-finger participation but affects contact posture differently.Doubling footprint weight yields a higher worst-tip planting fraction than doubling lift weight: paired difference +0.22 [+0.11, +0.33] in 10 of 12 seeds.
e) Contact posture and reward weights: Above the deployed footprint weight, worst-tip planting fraction keeps improving while speed changes remain uncertain; mean participation peaks at twice the deployed weight and declines by 3.3 times.Higher weights were tested only in simulation.

B. Untethered crawling and steering

The hand demonstrates untethered crawling across varied surfaces and learns steering with asymmetric turn responses. Closed-loop heading control compensates drift but remains limited for large left turns.

Untethered crawling: The hand crawled untethered across 14 indoor and outdoor surfaces, including metal grating, grass, and gravel.These runs are qualitative demonstrations shown in supplementary video.
Open-loop steering: Across 21 open-loop trajectories, mean path speed was 0.093 m/s, while the hand drifted right by about 6°s−1 without steering.Right-turn rate scaled with command, whereas left-turn rate plateaued.
Closed-loop steering: A proportional controller using onboard IMU heading added an offset to cancel drift and produced commanded heading changes in both directions.The controller kept yaw-rate commands within the training range.
Closed-loop steering: The hand reached ±15° target headings and a 30° right turn, but 30° left turns fell short because drift compensation pushed commands beyond the limit.This bounds the demonstrated closed-loop steering range.

C. Fall recovery

A learned recovery policy enables the hand to right itself after falls and return to its crawl stance without assistance.

Fall recovery: The hand recovered from 21 of 25 hardware falls, achieving an 84% recovery rate across thumb-side and wrist-side starting poses.It recovered from 11 of 14 thumb-side falls and 10 of 11 wrist-side falls.
Fall recovery: After righting, an upright-state detector triggered a smooth transition to the crawl stance, ready for locomotion.The four failures occurred when fingers caught on each other and the hand stalled.

D. Keyboard pressing

The hand performs self-supported keyboard interaction while maintaining its body support, and the broader hardware demonstrations combine interaction with locomotion and recovery. The current system remains constrained by heading range and reliance on external perception for some tasks.

Keyboard pressing: The keyboard policy produced 29 correct presses from 32 consecutive commands over 72.5 s without keyboard realignment.Commands were issued in mixed order on a fixed, manually aligned keyboard.
Keyboard pressing: The same interface successfully executed optimal 9- and 12-move Sokoban solutions for one- and two-box levels.The keystrokes were generated through physical keyboard pressing.
System scope: Self-supported interaction is part of a hardware platform that also demonstrates untethered locomotion, recovery, and vision-guided object pushing.The hand reuses its fingers for body support and environmental interaction.
Limitations and extensions: The demonstrated operating range could be broadened by extending commanded heading changes and adding onboard perception.Suggested additions include automated keyboard alignment and onboard hand and object tracking beyond the overhead camera’s workspace.