Cat-like Jumping and Landing of Legged Robots in Low-gravity Using Deep Reinforcement Learning

• Extended flight phases make ground-reaction forces alone impractical for attitude control, while reaction wheels add weight, mechanical complexity, and actuation requirements.The paper notes that take-off momentum can create attitude errors during flight that need correction to avoid crash landings.Source: §I · p. 1
• The authors use limb motion for attitude control, aiming to enable precision jumping and landing without relying on reaction wheels.The approach is inspired by animals and astronauts reorienting themselves with their limbs.Source: §I · p. 1
• The paper identifies nonlinear dynamics, self-collision avoidance, hard-contact discontinuities, fast reorientation, and immediate reactions as challenges for limb-only attitude control,,,,.Prior approaches discussed include optimization, repeated cyclic motions, and genetic algorithms; the paper notes limitations including computation, slow maneuvers, or a simple 2D demonstration.Source: §I · pp. 1–2
• In a two-dimensional pitch task, the deployed learned policy reaches its target faster and more precisely than handcrafted leg motions.The paper also describes a wall landing-and-jump task followed by a flip and landing on the opposite wall, tested on a two-dimensional microgravity testbed.Source: §I · p. 2
• In simulation, the policy controls all three axes and reaches target orientations from any initial state despite the robot’s two-degree-of-freedom legs; adding simulated abduction yields high performance on all axes.The most challenging task has the robot reorient during flight and land softly on a low-gravity body.Source: §I · p. 2

A. Robotic platform

• SpaceBok’s parallel, closed-chain legs support explosive force generation, but their configuration is difficult to simulate accurately.The robot has two actuators per leg, moves its feet in the sagittal plane, and has a maximum output torque of 39.5 Nm.Source: A. Robotic platform · p. 2

B. Simulation

• Training on the physical robot is impractical, so the policy is trained in simulation for transfer to hardware; simulation must balance speed with accuracy for contact and transfer.The paper uses domain randomization to handle simulation-to-reality mismatches.Source: B. Simulation · p. 3

1) Simulator selection:

• Mujoco with RK4 is selected because it conserves angular momentum in the reorientation task, whereas Euler integration diverges; RaiSim handles hard contacts and can be faster.The paper reports that RaiSim’s Euler method diverges even at small time steps, while Mujoco with RK4 conserves angular momentum.Source: 1) Simulator selection · p. 3

2) Closed kinematic chain simulation:

• Because standard simulators do not directly support SpaceBok’s closed chain, training uses a simplified shoulder-and-virtual-elbow configuration, with torques transformed for the physical parallel legs.Mujoco constraints impair angular-momentum conservation, while RaiSim does not allow closed-chain simulation.Source: 2) Closed kinematic chain simulation · p. 3

• Motor torque is clipped to its velocity-dependent allowable range, and torques are prevented from accelerating joints above a safe velocity threshold.The actuator model also clips controller torques to the allowed torque-velocity region.Source: 3) Actuator model · pp. 3–4
• The policy’s action space is desired joint positions, which a joint-space PD controller converts into actuator torques.Adjusting PD gains between simulation and deployment improves tracking without retraining the policy.Source: 3) Actuator model · p. 4

4) Self-collisions:

• Virtual foot-centered spheres trigger collision penalties during training, keeping feet farther apart from the legs and each other for safer deployed motion.Collisions terminate the episode with a large negative reward.Source: 4) Self-collisions · p. 4

1) Reinforcement Learning:

• The agent selects actions from observations using a neural-network Gaussian policy, whose weights are trained to maximize expected return.The policy learns both the mean and diagonal covariance; the mean depends on observations while the covariance is an independent learned variable.Source: 1) Reinforcement Learning · p. 4

2) Policy optimization:

• Policies use three 128-neuron hidden layers and about 35,000 parameters, trained with PPO for 150 million steps.Without domain randomization, training can be up to three times faster.Source: 2) Policy optimization · p. 4

3) Sim-to-Real transfer:

• Domain randomization varies foot weights, actuator friction, and PD gains across parallel simulated environments to address mismatches with the physical robot.The foot weights are randomized around 330 g by ±30%.Source: 3) Sim-to-Real transfer · p. 4

4) Deployment:

• After transfer, joint targets and torques are constrained to hardware limits, and observations are clipped; these limits can be adjusted without retraining.The network weights are transferred without retraining or domain adaptation.Source: 4) Deployment · p. 4

D. Experimental Set-Up

• The testbed simulates two-dimensional microgravity by letting the robot move in its sagittal plane on an air-bearing platform and rotate about its pitch axis.A motion-capture system measures base position, orientation, and velocities for the policy.Source: D. Experimental Set-Up · pp. 4–5

III. 2D ATTITUDE CONTROL

• For pitch control, the robot moves from a randomized initial orientation to a target, represented to the policy as an angle error to reduce to zero.The task has no ground contact.Source: §III · p. 5

1) Observation space:

• Observations contain joint positions and velocities plus base pitch angle and angular velocity, measured at the last time step without history.Source: 1) Observation space · p. 5

2) Action space:

• The policy outputs desired joint positions, with individual leg control compared against shared left-right trajectories and handcrafted motions.Desired positions and corresponding torques are processed to respect the closed-chain and actuator limits.Source: 2) Action space · p. 5

3) Reward:

• The reward prioritizes reducing pitch error while penalizing base acceleration, invalid joint configurations, and torques beyond actuator limits.Clipping penalties apply when actions or requested torques exceed allowed values.Source: 3) Reward · p. 5

A. Training

• Training uses a curriculum that gradually increases initial-state complexity, larger update batches and steps, an absolute orientation-error penalty, and a discount factor of γ = 0.9995.These changes address failures such as stopping short of the target or turning in only one direction.Source: A. Training · pp. 5–6

B. Simulation performance

• Figure 6 shows both learned policies outperforming handcrafted baselines; the asymmetric policy rotates 90° in less than 1.5 s, versus more than 3 s for baselines.Asymmetric motions also produce fewer back-and-forth movements and smoother trajectories.Source: B. Simulation performance · p. 6, Figure 6

1) Effect of feet weights:

• Figure 7 shows good performance from 30 g to 730 g without retraining, close to policies trained at the correct weight; flip performance plateaus around 600 g.Weights between 200 g and 400 g are described as a reasonable choice balancing performance and added robot mass.Source: Figure 7 · p. 6, 1) Effect of feet weights · p. 6

C. Deployment performance

• In simulation and on the physical robot, 90° rotation and stabilization take less than 2.5 s, and the deployed policy recovers from manual pushes despite not being trained for them.The comparison uses matched initial conditions and targets.Source: C. Deployment performance · pp. 6–7, Figure 8 · p. 7

IV. 2D JUMPING

• The jumping task starts the robot a few meters from a wall with up to 1 m/s perpendicular velocity and up to 180° orientation, then requires reorientation, landing, and a directed jump back.Source: §IV · p. 7

1) Observation space:

• Jump-task observations extend the earlier inputs with base height, base linear velocity, and desired base velocity.Source: 1) Observation space · p. 7

2) Action space:

• The jumping policy constrains left-right leg actions to produce symmetric forces during flight.Source: 2) Action space · p. 7

3) Reward:

• The jumping reward adds tracking of commanded jump direction and velocity plus a torque penalty to smooth landing actions, while increasing the base-acceleration penalty.The added reward terms have coefficients c5 = 1 and c6 = 1/30; the base-acceleration coefficient is c2 = 1/20.Source: 3) Reward · p. 7

A. Training

• Training applies a curriculum to initial and target base velocities for the two-dimensional jumping task.The examples include a robot approaching the ground at 1 m/s before reorienting and jumping toward a target velocity.Source: A. Training · p. 7, Figure 9 · p. 7

1) Simulation performance:

• 100 consecutive wall-to-wall jumps were completed without failure in simulation.The policy also reorients, lands, and jumps back in a given direction, including from complex states and after imperfect jumps.Source: 1) Simulation performance · p. 7

B. Deployment performance

• On the physical robot, jump-velocity error rose to ±25%, versus below ±8% in simulation.The authors attribute the gap most probably to imperfect modeling of friction between the feet and walls; they propose friction randomization and contact-model tuning as possible improvements.Source: B. Deployment performance · p. 8
• Dozens of consecutive jumps between walls were completed without failure on the physical robot.The policy selects a rotation direction based on post-jump conditions and can use the landing wall to correct incomplete rotations.Source: B. Deployment performance · p. 8, Figure 11 · p. 8

V. 3D ATTITUDE CONTROL

• Without hip abduction, a 90° roll or yaw rotation took over 6 s, reflecting reduced control authority on those axes.The simulated robot was extended with hip abduction/adduction actuators to pursue more practical 3D reorientation performance.Source: §V · p. 8

1) Observation space:

• Three-dimensional orientation is represented by a quaternion, with angular velocity represented as a three-dimensional vector.The policy observes and controls all legs individually because the 3D task has no left-right leg symmetry.Source: 1) Observation space · p. 8, 2) Action space · p. 8

3) Reward:

• The 3D orientation reward penalizes the angle from the angle-axis representation of the quaternion.Source: 3) Reward · p. 8

A. Training

• Three-dimensional training is unstable and requires a slower curriculum beginning with very small orientation errors.After about 50% of training steps, reward rises and a solution is found; the initial-state range is then expanded until the policy handles all possible orientations.Source: A. Training · p. 8

B. Simulation performance

• With added actuators, the policy controls all axes and performs any rotation in less than 3 s.Compared with the original leg configuration, the added actuator lets the policy reorient the robot four times faster.Source: B. Simulation performance · pp. 8–9, Figure 13 · p. 9

VI. 3D LANDING

• The landing task extends 3D reorientation to a robot that must land from a random orientation and remain on the ground.Its observations use quaternion orientation, and its action space controls all legs independently with added hip abduction/adduction actuators.Source: §VI · p. 9, 1) Observation space · p. 9, 2) Action space · p. 9

3) Reward:

• The 3D landing reward uses the 3D orientation term and sets the jumping target velocity to zero.Source: 3) Reward · p. 9

A. Training

• The landing curriculum still starts slowly at η = 0.25, but its reward oscillates less than in the orientation task.The authors suggest that crashing when reorientation fails or proceeds in the wrong direction gives a strong incentive to learn correct reorientation.Source: A. Training · p. 9

B. Simulation performance

• With only 3 s before impact, the policy prioritizes suitable foot placement over a perfect target orientation and can extend a lower leg to reorient just before landing.The examples include landings from initial roll and yaw orientations shown in Figure 14.Source: B. Simulation performance · p. 9, Figure 14 · p. 9
• Without terrain measurements or changes to observations, actions, or rewards, the policy landed on various uneven terrains.The policy was trained on uniform valleys up to 0.7 m deep and generalized to a steep crater 1.5 m deep.Source: B. Simulation performance · p. 9, Figure 15 · p. 9

A. Training process analysis

• In 3D reorientation, reward oscillated for up to 60 million steps, and a solution appeared only after 100 million steps, or 4 hours of computation.During the oscillatory period, reward stayed below the −0.4 do-nothing or random-leg-motion baseline.Source: A. Training process analysis · p. 10
• Jumping and landing training did not show the reward drops and oscillations seen in reorientation.The authors suggest that the high penalty for crashing when incorrectly oriented provides a useful incentive for smooth training.Source: A. Training process analysis · p. 10

B. On-board state estimation

• The work assumes orientation and ground-distance estimates are available, uses Vicon on hardware, and leaves onboard sensing and state estimation out of scope.Inertial and exteroceptive sensor fusion is suggested, but development depends on the target environment and its lighting, mass and power limits, and available computation.Source: B. On-board state estimation · p. 10

C. Terrain shape

• Current tasks use perfectly flat, rigid terrain, although blind uneven-terrain landing is reported; terrain observations are proposed for better performance and robustness.The paper notes that reinforcement learning can accept different observation types.Source: C. Terrain shape · p. 10

D. Comparison to Model-Based Approaches

• End-to-end learning can exploit ground contacts for last-second reorientation corrections, which splitting landing and jumping into separate solutions would miss.The authors also describe safety margins and risk penalties during training, plus joint-position and torque constraints during deployment.Source: D. Comparison to Model-Based Approaches · p. 10

VIII. CONCLUSIONS AND FUTURE WORK

• 90° pitch changes in under 2.5 s were achieved in simulation and on the physical robot with the learned policy.Two-dimensional hardware tests showed consecutive jumps without human intervention; three-dimensional and arbitrarily shaped terrain results are in simulation. Real-world 3D deployment remains future work and requires three degrees of freedom per leg and a replicated low-gravity environment.Source: §VIII · p. 11