PaperlayerView exact revision on arXiv

Source-linked AI summary

RMA: Rapid Motor Adaptation for Legged Robots

Ashish Kumar, Zipeng Fu, Deepak Pathak, Jitendra Malik

arXiv:2107.04034v1cs.LGcs.AIcs.CVcs.RO

TL;DR

Legged robots need rapid adaptation to unseen terrains and payloads, but real-world deployment is hindered by changing dynamics and unavailable environmental parameters. RMA combines a simulation-trained base policy with an adaptation module that estimates useful extrinsics online, achieving robust locomotion across varied conditions while remaining limited by failures under larger perturbations and the absence of vision.

  • Problem

    Legged robots must adapt in real time to changing terrains, payloads, wear, and other unseen conditions despite unavailable environmental parameters and a difficult sim-to-real gap.

  • Method

    RMA trains a base policy and an adaptation module entirely in simulation, then estimates extrinsics from recent state-action history to adapt the policy during deployment.

  • Results

    RMA achieves high success across varied indoor and simulation conditions, carries up to 12Kg with a high success rate, and performs close to the Expert baseline in simulation.

  • Takeaways & Limitations

    Rapid online extrinsics estimation enables the robot to adapt quickly without per-situation data collection, demonstrations, or predefined motion templates.

  • Takeaways & Limitations

    Using only proprioception, RMA sometimes fails under larger perturbations such as sudden falls or multiple leg obstructions from rocks.

Abstract

from arXiv · show

Successful real-world deployment of legged robots would require them to adapt in real-time to unseen scenarios like changing terrains, changing payloads, wear and tear. This paper presents Rapid Motor Adaptation (RMA) algorithm to solve this problem of real-time online adaptation in quadruped robots. RMA consists of two components: a base policy and an adaptation module. The combination of these components enables the robot to adapt to novel situations in fractions of a second. RMA is trained completely in simulation without using any domain knowledge like reference trajectories or predefined foot trajectory generators and is deployed on the A1 robot without any fine-tuning. We train RMA on a varied terrain generator using bioenergetics-inspired rewards and deploy it on a variety of difficult terrains including rocky, slippery, deformable surfaces in environments with grass, long vegetation, concrete, pebbles, stairs, sand, etc. RMA shows state-of-the-art performance across diverse real-world as well as simulation experiments. Video results at https://ashish-kmr.github.io/rma-legged-robots/

I. INTRODUCTION

Legged robots must adapt online to changing physical conditions despite a difficult sim-to-real gap. RMA addresses this with a simulation-trained base policy and adaptation module that infer useful environmental representations from recent proprioceptive history.

  • The sim-to-real gap reflects mismatches in robot models, real terrains, and simulator physics, especially for contact forces and deformable surfaces.
  • RMA’s asynchronous deployment runs adaptation at 10Hz and the base policy at 100Hz, using the latest extrinsics prediction for action generation.
  • RMA targets online adaptation within fractions of a second as robots encounter different soils, slopes, payloads, fatigue, and disturbances.
  • The base policy is trained with privileged environmental factors encoded into extrinsics and learns actions from state, previous action, and those extrinsics.
  • The adaptation module estimates extrinsics from recent state-action history using supervised learning on simulation data, replacing unavailable privileged information at deployment.
  • RMA avoids per-situation data collection and takes less than 1s, compared with 4−8mins for Peng et al.’s approach.
  • The method uses varied terrain generation and bioenergetics-motivated natural rewards to learn walking without reference demonstrations.

II. RELATED WORK

Prior legged-locomotion approaches use control, reinforcement learning, domain randomization, simulation refinement, or online system identification. These methods trade modeling effort, robustness, optimality, or deployment data requirements in different ways.

  • Control-based locomotion methods rely on modeled dynamics, parameterized controllers, and prior expertise to plan or optimize movement.
  • Deep reinforcement learning reduces reliance on human expertise and performs well in simulation, but learned policies remain difficult to transfer to the real world.
  • Domain randomization trains across environment parameters and sensor noise, but robustness can come at the cost of an over-conservative policy.
  • More accurate simulation can reduce the sim-to-real gap, but fitted motor models require initial robot data and must be refit for every new setup.
  • Online system identification conditions policies on inferred physics parameters, although exact parameter prediction is often difficult and unnecessary.

A. Base Policy

The base policy uses reinforcement learning to map robot state, previous action, and an encoded environment vector to joint-position actions. Training combines bioenergetics-inspired rewards, uneven terrain, and a curriculum over penalties and perturbations.

  • A. Base Policy: The base policy takes current state x_t, previous action a_t−1, and extrinsics z_t to predict the next action a_t.The action is a desired position for the 12 robot joints, converted to torque by a PD controller.
  • A. Base Policy: The environment encoder maps the 17-dimensional environment vector e_t into an 8-dimensional extrinsics representation z_t.
  • A. Base Policy: Training jointly optimizes the base policy and environment encoder with model-free reinforcement learning using expected discounted return.The trajectory distribution under policy π defines the expectation being optimized.
  • A. Base Policy: Bioenergetics-inspired penalties for work and ground impact, together with uneven-terrain training, encourage realistic and robust gaits without reference demonstrations.The reward also promotes forward motion while penalizing lateral movement, rotation, jerky motion, and inefficiency.
  • A. Base Policy: A fixed curriculum gradually increases penalty coefficients and perturbation difficulty to prevent the policy from learning to remain stationary.

B. Adaptation Module

The adaptation module estimates environment-dependent extrinsics online from recent state-action history, replacing privileged environment information unavailable during real-world deployment. It is trained in simulation with supervised targets and on-policy data to remain robust to imperfect estimates.

  • B. Adaptation Module: The adaptation module estimates the extrinsics vector from the robot’s recent states and actions instead of using privileged environment information.The experiments use a 50-step history corresponding to 0.5 seconds.
  • B. Adaptation Module: RMA estimates extrinsics directly because they encode how behavior should change, rather than reconstructing the full environmental parameter vector.
  • B. Adaptation Module: The module is trained by minimizing mean squared error between predicted extrinsics and encoder-generated targets available in simulation.The adaptation module is implemented as a 1-D CNN that captures temporal correlations.
  • B. Adaptation Module: Training on-policy exposes the adaptation module to imperfect and exploratory trajectories rather than only seamless trajectories from the ground-truth policy.The policy is iteratively unrolled using predictions from a randomly initialized adaptation module, then paired with ground-truth extrinsics for training.

C. Asynchronous Deployment

RMA separates fast base-policy control from slower adaptation updates, enabling asynchronous deployment on the A1 robot. The deployed system is trained only in simulation and evaluated across varied environmental conditions and real-world terrains.

  • C. Asynchronous Deployment: The adaptation module updates the extrinsics estimate at 10 Hz, while the base policy runs at 100 Hz using the latest estimate.The two subsystems therefore operate asynchronously without requiring synchronization or calibration.
  • C. Asynchronous Deployment: The decoupled design avoids the unnatural gaits, poor simulation performance, and 10 Hz compute limit found when state-action history is fed directly into one policy.
  • C. Asynchronous Deployment: RMA is deployed on the 12 kg, 12-actuated-degree-of-freedom A1 robot using onboard joint, IMU, and foot-contact measurements.
  • C. Asynchronous Deployment: The simulation varies mass, mass position, motor strength, friction, and local terrain height within a 17-dimensional environment vector.Terrain height is summarized from the maximum discretized height beneath the four feet.
  • C. Asynchronous Deployment: RMA steps down 15 cm with 80% success and crosses unseen deformable surfaces with 100% success, while also climbing inclines and steps.The reported figures are averaged over 5 trials.

B. Training Details

The implementation uses multilayer perceptrons for the base policy and environment encoder, and a temporal convolutional adaptation module. Separate PPO and supervised-learning procedures train these components in simulation.

  • B. Training Details: The base policy is a 3-layer MLP that maps a 30-dimensional state, 12-dimensional previous action, and 8-dimensional extrinsics vector to 12 target joint angles.
  • B. Training Details: The environment factor encoder is a 3-layer MLP that encodes the 17-dimensional environment vector into an 8-dimensional extrinsics vector.
  • B. Training Details: The adaptation module embeds states and actions, applies a 3-layer 1-D CNN across time, and linearly projects the result to estimate extrinsics.
  • B. Training Details: PPO jointly trains the base policy and environment encoder for 15,000 iterations, simulating 1.2 billion steps in roughly 24 hours.
  • B. Training Details: Supervised learning trains the adaptation module with on-policy data for 1,000 iterations, simulating 80 million steps in roughly 3 hours.
  • B. Training Details: On an oily plastic surface, RMA detects slip through changes in predicted extrinsics, increases torque, and recovers the gait period after adaptation.RMA succeeds in 90% of runs over the oily patch.

V. RESULTS AND ANALYSIS

RMA is evaluated against manufacturer, robustness, expert, system-identification, and adaptation baselines using simulation and indoor real-world tests. It achieves high success across indoor setups, with adaptation especially important for forward locomotion and payload handling.

  • Evaluation setup: RMA is compared with the A1 controller, Robust, Expert, RMA without adaptation, System Identification, and AWR baselines.The evaluation uses normalized time-to-fall, forward reward, success rate, distance, adaptation samples, torque, smoothness, and ground impact.
  • Indoor experiments: Indoor real-world comparisons use the A1 controller and RMA without adaptation to limit potential hardware damage.Trials cover payloads, steps, uneven foam, mattresses, inclines, and oily surfaces.
  • Indoor experiments: RMA achieves a high success rate across indoor setups and substantially outperforms the A1 controller in some cases.The A1 controller struggled with uneven foam and large step transitions, while unstable footholds contributed to its failures.
  • Indoor experiments: RMA carries up to 12Kg, equal to 100% of body weight, while maintaining height and a high success rate as payload increases.The A1 controller began sagging and eventually fell as payload increased; RMA without adaptation mostly did not fall but failed to move forward.
  • Indoor experiments: Turning off adaptation substantially degrades performance, indicating that the adaptation module is critical for these indoor tasks.On an oily path with plastic-wrapped feet, RMA successfully crossed the slippery patch.

B. Outdoor Experiments

RMA is tested outdoors on deformable, obstructed, sloped, and debris-covered terrain. It succeeds consistently on several difficult surfaces, while stair and construction-debris performance is lower but remains substantial.

  • Outdoor environments: RMA walks on sand, mud, and dirt without a single failure across all reported trials.These terrains require dynamically changing footholds because feet can sink or stick.
  • Outdoor environments: RMA achieves a 100% success rate on tall vegetation and while crossing a bush.The robot stabilizes against foot entanglements and powers through obstructions.
  • Outdoor environments: RMA succeeds in 70% of trials walking down stairs despite never seeing a staircase during training.The stair task was conducted on a hiking trail.
  • Outdoor environments: RMA succeeds 100% of the time downhill over a mud pile and 80% of the time across cement and pebble piles.The cement and pebble piles were on ground that sloped steeply sideways.

C. Simulation Results

Simulation and adaptation analyses show that RMA performs best among the tested baselines and rapidly responds to slipperiness using proprioceptive history. The conclusion also identifies a boundary: larger perturbations can still cause failures without vision.

  • Simulation results: RMA outperforms all simulation baselines, with only slight degradation relative to the Expert.Results average three randomly initialized policies over 1000 episodes per initialization with environment resampling during testing.
  • Simulation results: AWR adapts slowly in changing environments, Robust learns a conservative policy, and System Identification performs poorly relative to RMA.RMA without adaptation also shows a significant performance drop.
  • Adaptation analysis: RMA succeeds in 90% of runs across an oily patch with plastic-wrapped feet.The analysis examines gait patterns, knee torque profiles, and estimated extrinsics during adaptation.
  • Adaptation analysis: When slipping begins, the estimated extrinsics change, enabling recovery and continued walking across the slippery patch.After adaptation, torque stabilizes at a slightly higher magnitude and the gait period roughly recovers, while the estimate continues to encode slipperiness.
  • Limitations: Without exteroception, larger perturbations such as sudden falls or multiple rock obstructions sometimes cause failures.The authors identify onboard vision as an important direction for more reliable walking.

S1. ADDITIONAL TRAINING AND DEPLOYMENT DETAILS

The training and deployment pipelines use simulation rollouts to train the base policy and encoder, then train the adaptation module to predict latent environment factors from recent state-action history.

  • Pipeline: The training pipeline is presented in Algorithm 1 and the deployment pipeline in Algorithm 2.The pipelines separate base-policy training from adaptation-module training and deployment.
  • Base-policy training: PPO trains the base policy and environmental factor encoder for 15,000 iterations using batches of 80,000 state-action transitions.The implementation uses four mini-batches and four update rounds per mini-batch.
  • Reward curriculum: Penalty terms are introduced gradually through a curriculum beginning at k_0 = 0.03 to prevent reward-driven training collapse.The coefficients increase exponentially according to k_t+1 = k_t 0.997.
  • Base-policy training: The base-policy rollout encodes environmental parameters into z_t, conditions actions on that latent representation, and stores transitions for PPO updates.The encoder and policy are updated from simulated environment interactions.

S1. ADDITIONAL REAL-WORLD ADAPTATION ANAYLSIS

The additional real-world analysis examines how RMA responds to a sudden 5 kg payload and how training rewards converge over 15,000 iterations. After the disturbance, the robot recovers its gait while adapting torque output to the added mass.

  • Payload adaptation: A 5 kg payload disturbs the robot’s regular motion, after which RMA enters an adaptation phase and recovers.
  • Payload adaptation: Post-adaptation, the gait period is roughly recovered while knee torque stabilizes at a higher magnitude to account for the payload.
  • Payload adaptation: The estimated extrinsics vector changes when the payload lands, indicating that the adaptation module detects the added mass.
  • Training behavior: Across 15,000 training iterations, average step rewards are shown for total reward, forward-plus-lateral reward, and penalty terms as they converge.
  • Training behavior: Applying a small initial multiplier to penalty terms prevents entirely negative early experience that would otherwise hinder rapid walking acquisition.

S2. ADDITIONAL SIMULATION TESTINGS

Additional simulations test RMA under variations in payload, terrain elevation, and foot–terrain friction. The accompanying payload experiment shows rapid behavioral adjustment, including recovered gait timing and increased torque after loading.

  • Simulation generalization: RMA is evaluated in simulation across payload, terrain-elevation, and foot–terrain-friction variations using success rate, TTF, and reward.
  • Simulation generalization: The simulation comparison reports RMA’s superiority over baseline methods across all tested variation types and metrics.
  • Payload adaptation: With a 5 kg payload added during a run, the robot’s center of mass dips before the adaptation module recovers from the disturbance.
  • Payload adaptation: After adaptation to the payload, the gait period is roughly restored, torque magnitudes increase, and the extrinsics estimate continues to represent the added load.
Loading 2107.04034v1…