Source-linked AI summary
Learning Arm-Assisted Fall Damage Reduction and Recovery for Legged Mobile Manipulators
Yuntao Ma, Farbod Farshidian, Marco Hutter
TL;DR
Legged mobile manipulators can damage their payloads and manipulators when they fall, and prior approaches often rely on restrictive assumptions. This paper trains a time-invariant policy with asymmetric actor-critic learning and time-varying rewards. The policy reduces fall-damage measures against emergency-controller baselines and is tested on ALMA hardware.
Problem
Fall damage threatens payloads and manipulators on legged mobile manipulators, while prior strategies often assume inelastic collisions or falls in defined directions.
Source: §I · p. 1Method
An asymmetric actor-critic trains a time-invariant fall-reduction and recovery policy using time-varying task rewards and privileged critic observations.
Source: §I · p. 1, B. Asymmetric actor-critic · p. 3, D. Reward function · p. 3Results
Takeaways & Limitations
The policy learns adaptive arm-assisted fall damage reduction and recovery, and was extensively tested in simulation and on ALMA hardware.
Source: §V · p. 6
Abstract
from arXiv · showhide
Adaptive falling and recovery skills greatly extend the applicability of robot deployments. In the case of legged mobile manipulators, the robot arm could adaptively stop the fall and assist the recovery. Prior works on falling and recovery strategies for legged mobile manipulators usually rely on assumptions such as inelastic collisions and falling in defined directions to enable real-time computation. This paper presents a learning-based approach to reducing fall damage and recovery. An asymmetric actor-critic training structure is used to train a time-invariant policy with time-varying reward functions. In simulated experiments, the policy recovers from 98.9\% of initial falling configurations. It reduces base contact impulse, peak joint internal forces, and base acceleration during the fall compared to the baseline methods. The trained control policy is deployed and extensively tested on the ALMA robot hardware. A video summarizing the proposed method and the hardware tests is available at https://youtu.be/avwg2HqGi8s.
I. INTRODUCTION
Legged mobile manipulators risk damage to payloads and manipulators when they fall, while existing fall strategies often rely on restrictive assumptions. This work uses learning to reduce fall damage and recover with arm assistance, and also adapts the method to resting and self-righting tasks.
I. INTRODUCTION
- Payloads such as sensors and end effectors, as well as the manipulator, are vulnerable to fall damage, making field deployment more challenging.Fall damage reduction and recovery remain key challenges in legged robotics.Source: §I · p. 1
- Prior controllers commonly assume inelastic, non-sliding collisions or constrain falls to the sagittal or frontal plane.Some methods also assume limbs can track adaptive contact plans, which may not hold for robots with heavier legs and restrictive joint-velocity limits.Source: §I · p. 1
- The approach trains one fall-recovery policy with an on-policy asymmetric actor-critic, time-varying rewards, and arm assistance, avoiding brittle tuning of motion-smoothness and recovery-time weights.It extends prior learning-based recovery methods,.Source: §I · p. 1
- The policy reduces peak base impulse, peak joint internal force, and base acceleration versus deployed freezing and damping emergency controllers.Ablations also report improved recovery behavior over symmetric and time-variant policy versions.Source: §I · p. 1
- The method is validated in simulation and on ALMA hardware for adaptive falling and arm-assisted recovery to stance within a specified time horizon.The robot resumes its stance configuration after recovery.Source: §I · p. 1
- The method is also adapted to bringing the robot to a resting configuration and to simple self-righting, with hardware validations of both policies.Source: §I · p. 2
II. RELATED WORK
Prior fall-damage methods select fixed or adaptive contact sequences and optimize impact-related criteria, but often simplify the fall or contact model. Fully stretched arm configurations are also undesirable for heavy robots such as ALMA because impacts stress the drives.
A. Fall damage reduction
- Safe-falling work formulates optimization around momentum change, impact force, acceleration, or combined damage criteria.The cited criteria include,,,,,, and.Source: A. Fall damage reduction · p. 2
- Earlier strategies prescribe a contact sequence, including forward-fall ukemi or tripod falls, with contact locations computed directly or selected using reinforcement learning.Source: A. Fall damage reduction · p. 2
- Ha and Liu use contact graphs to minimize vertical instantaneous impulse with a predefined contact sequence; Kumar et al. use mixture-of-experts to select the next contact body during a fall.Multiple-contact planning methods restrict falls to the sagittal plane, and robot falls may assume inelastic contact.Source: A. Fall damage reduction · p. 2
- A fully stretched arm can stop small humanoids quickly, but the impact stress it induces on drives makes that configuration undesirable for heavy robots such as ALMA, approximately 58 kg.Source: A. Fall damage reduction · p. 2
B. Fall recovery
Fall-recovery methods face uncertain state and contact information, while optimization-based approaches often depend on heuristics and robot-specific engineering. The paper uses learning-based recovery with randomized falling initial states and delayed actuator activation.
B. Fall recovery
- Optimization-based recovery often relies on predefined control sequences, specified state transitions, or model simplifications, and these heuristics require extra engineering to transfer across platforms.Unreliable state estimation and uncertain contacts after a fall further challenge optimization-based control.Source: B. Fall recovery · p. 2
- Learning-based recovery is presented as relying less on heuristics and simplifications; prior RL methods formulate self-righting as an infinite MDP and use a behavior selector to combine self-righting, standing, and locomotion.K-Access uses state-space clustering to improve learning sample efficiency; reward tuning remains a challenge that this work addresses with time-varying rewards.Source: B. Fall recovery · p. 2
III. METHOD
- Training uses a finite-horizon MDP with randomized falling configurations and rewards for damage reduction and recovery to stance; an asymmetric actor-critic gives the critic noiseless privileged observations.The actor observes minimally, while the critic uses privileged information to improve value estimation; the policy output becomes joint-position targets for the PD controller.Source: §III · p. 2, Figure 2 · p. 2
A. State initialization and rollout
- Initial base and joint states are randomized, and actuators are disabled for a random 0.04–1.50 seconds to model delayed fall detection and challenging initial conditions.Episodes continue through the MDP time horizon, after which the policy controls the robot.Source: A. State initialization and rollout · p. 3
B. Asymmetric actor-critic
The asymmetric actor-critic trains a deployable actor using privileged simulation-only information for critic value estimation. The actor outputs joint-position targets based on scaled actions around default joint positions.
B. Asymmetric actor-critic
- The critic receives noiseless actor observations plus privileged features unavailable at deployment, including remaining episode time and foot-contact states, while the actor receives noisy robot-state observations.Base linear velocity is excluded from the actor observation because of high state-estimation uncertainty after a fall.Source: B. Asymmetric actor-critic · p. 3
- Only the critic observes remaining episode time, allowing value estimation for the finite-horizon MDP while the actor policy remains time-invariant.The limited actor observation constrains policy structure rather than changing the optimized MDP.Source: B. Asymmetric actor-critic · p. 3
C. Actions
- The policy's joint-position target is the default joint position plus a scaled action output, and those targets are sent to the drives' PD controller.The default pose and action scale are chosen so random actions can sometimes flip the robot during sideways falls, supporting early self-righting exploration.Source: C. Actions · p. 3
- The reward-term table summarizes the terms used by the method.Source: Table I · p. 3
D. Reward function
The reward function combines time-varying recovery objectives with time-invariant penalties for damaging or undesirable behavior. Recovery rewards are active only in the final 2.0 seconds; both reward groups are zero while actuation is disabled.
D. Reward function
- Fall damage reduction minimizes undesired measurements such as contact impulse and body acceleration, while rewarding an end-of-episode return to a stance near ALMA’s default pose.Recovery task rewards activate only in the last 2.0 seconds; behavior penalties for collision, momentum change, and body yank remain active throughout training.Source: D. Reward function · p. 3
- The recovery task rewards higher torso height, penalizes deviation from ALMA’s default joint positions, and reward recovery of base orientation by penalizing roll and pitch.The maximum height reward applies at heights of at least 0.5 m.Source: D. Reward function · p. 3
- Task and behavior-regularization rewards are both zero during the initial phase when actuation is disabled, because the policy cannot affect joint actions then.The reward definitions specify joint position, velocity, and acceleration; projected gravity; selected robot links; and body mass, acceleration, contact force, and net force.Source: D. Reward function · p. 4
E. Sim-to-Real
The sim-to-real setup uses terrain, observation, robot, and actuator variations to support transfer from simulation to the ALMA hardware. The simulator runs at 200 Hz and the policy at 100 Hz.
E. Sim-to-Real
- The training simulator runs at 200 Hz, with the policy decimated to run at 100 Hz.The paper introduces these techniques to facilitate sim-to-real transfer.Source: E. Sim-to-Real · p. 4
- The simulator models leg-drive actuators, while arm drives receive randomized friction and torque delays without an actuator model.The arm’s pseudo-direct drives have better transparency than the legs’ series elastic actuators.Source: E. Sim-to-Real · p. 4
- Uneven terrain randomizes ground-contact normal directions and encourages greater clearance when the robot moves above the ground..Source: E. Sim-to-Real · p. 4
- Training adds Gaussian noise to actor observations and randomizes base mass and body-ground friction coefficients.Additional base mass is sampled uniformly from −5 to 5 kg in each episode.Source: E. Sim-to-Real · p. 4
IV. RESULTS
Across randomized fall tests, the learning policy reduced base impact and acceleration relative to freezing and damping controllers without incurring higher peak joint internal forces.
A. Fall damage reduction
- The learning policy significantly lowered 95th-percentile base acceleration than the freezing and damping controllers, while avoiding base contact impulses above 0.05 Ns more often.The comparison used 2,560 test rollouts with randomized initial fall configurations and base mass.Source: A. Fall damage reduction · p. 4
- The proposed policy marginally improved peak joint internal-force distributions over damping and the fall-recovery controller, and significantly reduced peak forces versus freezing.Overall, it reduced base contact impulse and acceleration without higher peak internal joint forces.Source: A. Fall damage reduction · p. 4
B. Fall recovery
The policy adapts arm use across fall recovery, including after a failed quick attempt, and improves recovery success over a tugged-arm baseline. Arm-assisted recovery also reduces average leg torque.
B. Fall recovery
- After a quick recovery attempt fails, the policy uses its arm and right-front knee to secure the fall, pushes against the ground with the arm, then retracts it and adjusts stance.The robot initially jumps with its right legs and balances by extending its left legs, but loses balance.Source: B. Fall recovery · p. 4, Figure 4 · p. 5
- The arm-assisted recovery policy succeeded in 98.9% of 2,560 test episodes, compared with 95.2% for the tugged-arm policy.Success required a final base height above 0.5 m and maximum joint velocity below 0.01 rad/s.Source: B. Fall recovery · p. 4
- Arm-assisted recovery reduced average torque consumption by 9.17% across all leg drives compared with recovery without arm help.Without the arm, the robot must tug its legs further to push and flip itself when falling sideways.Source: B. Fall recovery · p. 5
C. Ablation for the observation configurations
The observation ablations show that privileged critic observations support learning, while keeping episode time out of the actor produces more temporally consistent recovery behavior. The non-privileged, time-invariant actor gives up a small amount of mean return relative to a privileged policy that cannot be deployed directly.
C. Ablation for the observation configurations
- The observation-configuration comparison reports mean episode return and critic value error, averaged over three seeds after 20,000 training iterations.The evaluation corresponds to 3.93 × 10^9 environment steps.Source: C. Ablation for the observation… · p. 5, Table II · p. 5
- The privileged critic has drastically lower value-estimation error and enables the non-privileged actor to learn fall-damage reduction and recovery.Without episode-progress or privileged robot-state observations, the critic configuration is unable to learn the task.Source: C. Ablation for the observation… · p. 5
- Including remaining episode time in the actor leaves episode returns largely unchanged but produces less uniform recovery than the time-invariant actor.The time-variant policy may rest on hip-drive joint limits before rapidly standing, and may stop trying to recover after a failed attempt; the time-invariant policy always attempts recovery. Both profiles complete recovery within the finite-horizon task, while the time-invariant profile is more uniform.Source: C. Ablation for the observation… · p. 5, Figure 8 · p. 6
- The non-privileged, time-invariant actor has a 3.0% decrease in mean episode return compared with the privileged policy, which requires further distillation for hardware deployment.The privileged policy requires privileged actor observations and therefore cannot be directly deployed on the robot.Source: C. Ablation for the observation… · p. 6
D. Reproducibility and adaptation to other tasks
The time-based reward scheme yields similar recovery profiles when task rewards are scaled up threefold. The training pipeline can be adapted to resting and self-righting by changing high-level settings, and both tasks produce natural contact sequences that reduce base impact while reaching their target configurations.
D. Reproducibility and adaptation to other tasks
- Tripling the time-variant task rewards produces a recovery height distribution similar to the original reward scale.Both policies recover before task rewards activate and optimize behavior rewards.Source: D. Reproducibility and adaptation to… · p. 6
- The training pipeline adapts to other state-transition tasks through changes to high-level settings, and the resting and self-righting policies were validated on hardware.Settings include rollout initialization period, task duration, target joint position, and reward scales.Source: D. Reproducibility and adaptation to… · p. 6
- For resting, randomized initial stance configurations are trained by sampling desired joint positions during initialization.On uneven terrain, directly resetting the simulator to random stance poses is difficult; the behavior rewards prioritize reducing joint velocity over torque consumption.Source: D. Reproducibility and adaptation to… · p. 6
- For self-righting, a 2.0 s initialization period ensures the robot falls completely, and the target is its default resting configuration.The policy self-rights from a fallen state to the default resting position on the floor.Source: D. Reproducibility and adaptation to… · p. 6
- For both tasks, the training pipeline discovers contact sequences that reduce peak base impact while reaching the desired final joint configuration.This applies to the resting and self-righting tasks described in the section.Source: D. Reproducibility and adaptation to… · p. 6
V. CONCLUSION
The paper presents an asymmetric actor-critic method for a time-invariant policy that uses the arm to reduce fall damage and assist recovery. The conclusion reports improvements over emergency-controller baselines on three damage measures and identifies unmodeled fall damage as a limitation.
V. CONCLUSION
- The policy improves over emergency-controller baselines in peak instantaneous impulse, base acceleration, and peak joint internal forces during falls.It also outperforms the learning-based arm-tugged recovery policy in recovery success rate and leg torque consumption, and was tested in simulation and on ALMA hardware.Source: §V · p. 6
- The asymmetric actor-critic method trains a time-invariant policy under randomized fall configurations and time-based rewards.The policy adaptively uses the arm for fall-damage reduction and recovery.Source: §V · p. 6
- The training environment avoids base-ground contact and does not adapt to fall damage such as dysfunctional drives.The authors identify adapting to such damage as future work.Source: §V · p. 6