PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Whole-Body Manipulation with a Large-Scale Soft Robotic Torso via Guided Reinforcement Learning

Curtis C. Johnson, Carlo Alessi, Egidio Falotico, Marc D. Killpack

arXiv:2509.23556v1cs.RO

TL;DR

Soft robots’ uncertain dynamics make it difficult to build simulations that are both accurate enough for transfer and fast enough for data-intensive learning. The paper combines a fast MuJoCo simulation with motion-primitive-guided RL, achieving zero-shot transfer to Baloo and analyzing the resulting policy’s reactive behavior and limitations.

  • Problem

    Soft robots’ nonlinear, uncertain dynamics complicate accurate modeling, and simulation must also be fast enough to support data-intensive learning.

    Source: §I · p. 2

  • Method

    The paper uses a MuJoCo simulation and guides reinforcement learning with a hand-designed grasping motion primitive.

    Source: §I · p. 2, 4) Reward Engineering for Whole-Body… · p. 7

  • Results

    An 88% hardware success rate was achieved in zero-shot sim-to-real transfer, and the learned policy showed reactive behaviors including re-grasping and perturbation recovery.

    Source: §I · p. 2

  • Takeaways & Limitations

    Guided learning transforms the open-loop primitive into a reactive policy, although the learned policy can also over-correct under perturbations.

    Source: §I · p. 2

    The method relies on a hand-designed motion primitive, so a new primitive must be designed for each new task.

    Source: §VI · p. 15

Abstract

from arXiv · show

Whole-body manipulation is a powerful yet underexplored approach that enables robots to interact with large, heavy, or awkward objects using more than just their end-effectors. Soft robots, with their inherent passive compliance, are particularly well-suited for such contact-rich manipulation tasks, but their uncertainties in kinematics and dynamics pose significant challenges for simulation and control. In this work, we address this challenge with a simulation that can run up to 350x real time on a single thread in MuJoCo and provide a detailed analysis of the critical tradeoffs between speed and accuracy for this simulation. Using this framework, we demonstrate a successful zero-shot sim-to-real transfer of a learned whole-body manipulation policy, achieving an 88% success rate on the Baloo hardware platform. We show that guiding RL with a simple motion primitive is critical to this success where standard reward shaping methods struggled to produce a stable and successful policy for whole-body manipulation. Furthermore, our analysis reveals that the learned policy does not simply mimic the motion primitive. It exhibits beneficial reactive behavior, such as re-grasping and perturbation recovery. We analyze and contrast this learned policy against an open-loop baseline to show that the policy can also exhibit aggressive over-corrections under perturbation. To our knowledge, this is the first demonstration of forceful, six-DoF whole-body manipulation using two continuum soft arms on a large-scale platform (10 kg payloads), with zero-shot policy transfer.

I. INTRODUCTION

The introduction motivates whole-body manipulation with soft robots for bulky, heavy objects, while noting that uncertain soft-robot dynamics make simulation and control difficult. It presents a fast MuJoCo simulation and motion-primitive-guided reinforcement learning, and positions the work among related approaches to soft-robot manipulation.

I. INTRODUCTION

  • Whole-body manipulation lets robots use more than end-effectors to handle bulky or heavy objects that are difficult to manipulate with hands alone.Soft robots’ passive compliance makes them well-suited to contact-rich manipulation of objects with uncertain size, geometry, or weight.Source: §I · p. 1
  • Soft robots’ nonlinear, hysteretic, uncertain dynamics make accurate modeling difficult, while small simulation inaccuracies can cause learned policies to fail on hardware.The resulting sim-to-real gap makes it challenging to build a simulation both accurate enough for transfer and fast enough for data-intensive learning.Source: §I · p. 2
  • The paper’s MuJoCo simulation is presented as both fast enough for data-intensive learning and accurate enough for zero-shot policy transfer.The introduction reports speeds up to 350x real time on a single thread.Source: §I · p. 2
  • An 88% hardware success rate was achieved for zero-shot transfer of a learned whole-body grasping policy, with motion-primitive guidance critical where standard reward shaping struggled.The simulation runs up to 350x real time on a single thread; the work also analyzes how guided learning can yield reactive behaviors and over-corrections.Source: §I · p. 2

II. RELATED WORK

  • Compared with, the work studies guided RL on a purely soft robot and analyzes how learned behavior differs from its motion primitive.The paper describes these results as complementary to.Source: §I · p. 2

1) Modeling and Simulation of Soft Robots:

The related work frames soft-robot simulation as a speed–accuracy tradeoff and describes the paper’s fast MuJoCo implementation. It also situates the work in soft-robot learning and multi-arm coordination, before describing the Baloo platform.

1) Modeling and Simulation of Soft Robots:

  • The MuJoCo lumped-parameter simulation reaches up to 350x real time, and its open-source implementation analyzes speed–accuracy tradeoffs.The approach adapts a serial chain of rigid bodies with spring-loaded joints; the cited SoMo simulation reaches up to 0.5x real time.Source: 1) Modeling and Simulation of Soft… · p. 3

2) Learning-based Control of Soft Robots:

  • Prior soft-robot learning includes simulated tasks and limited real-world transfer, while this work investigates large-scale whole-body manipulation that lifts and supports heavy objects in all six DoFs.The paper also explores an alternative to full expert demonstrations, which are challenging to gather and map to deformable continuum robots.Source: 2) Learning-based Control of Soft Robots · p. 3

3) Coordinated Control of Multiple Soft Robots:

  • Prior multi-arm soft-robot work includes payloads up to 272 grams, but the authors report no prior forceful, contact-rich multi-arm coordination of significant loads in all six DoFs.Source: 3) Coordinated Control of Multiple… · p. 3

A. Hardware Description

  • Baloo has two pneumatically actuated arms, each with three soft continuum joints; joint configuration is represented with two bending DoFs and no twisting DoFs.The torso is mounted on a linear actuator, and VIVE trackers estimate joint configuration and measure the base and box poses.Source: A. Hardware Description · pp. 3–4

B. Simulation Description

The simulation represents soft continuum joints as serial disks connected by universal joints, with stiffness, damping, actuation, and joint limits chosen to approximate the hardware. Its configuration reporting uses a two-DoF constant-curvature assumption that can fail under forceful shearing.

B. Simulation Description

  • Serial rigid disks connected by universal joints approximate each continuum joint, with the MuJoCo implementation using N = 10 disks and spatial tendons for actuation.The disks are evenly spaced and share the joint’s mass; each universal joint provides two degrees of freedom.Source: B. Simulation Description · p. 4, Figure 5 · p. 5

1) Soft Continuum Joints:

  • The forward kinematics compose the transformations of serial universal-joint segments to approximate the continuum shape.Figure 4 compares two- and four-disk approximations with a true continuum joint in constant- and non-constant-curvature cases.Source: 1) Soft Continuum Joints · pp. 4–5, Figure 4 · p. 5
  • The model reports two joint configuration positions and velocities from the base-to-tip orientation, assuming constant curvature; forceful interactions can violate this assumption through shearing.The simulator tracks universal-joint angles internally but abstracts them away as on the hardware.Source: 1) Soft Continuum Joints · p. 5
  • Equal springs and dampers in series across the disks set the modeled joint stiffness and damping from the overall joint values.The hardware’s high torsional stiffness motivates using universal joints rather than ball joints, avoiding extra torsional tuning and computation.Source: 1) Soft Continuum Joints · p. 5
  • Three joint-limit approaches are described: per-joint hard limits, actuation balanced against passive stiffness, and physical contact between adjacent disks.The implementation combines all three, emphasizing hard limits and disk contact; disk thickness must be tuned for different disk counts.Source: 1) Soft Continuum Joints · pp. 5–6

2) Actuation:

Pneumatic actuation is modeled through spatial tendons and stateful cylinder actuators, while rigid components use hardware-based mass properties and a separately planned elevator trajectory.

2) Actuation:

  • Spatial tendons pass through four chamber-representing points per disk, while stateful cylinder actuators apply repulsive forces that bend the joints.Including pneumatic pressure state makes the overall model third order and lets actuation be reasoned about in pressure space, closer to the hardware than direct joint torques.Source: 2) Actuation · p. 6

3) Rigid Components:

  • Rigid links use the masses of the hardware housed within them and are approximated as solid cylinders with centroidal centers of mass.Their inertia matrices are aligned with the cylinders’ principal axes.Source: 3) Rigid Components · p. 6
  • The torso elevator uses smooth Ruckig-planned hardware trajectories, replicated in MuJoCo with a custom plugin.Source: 3) Rigid Components · p. 6

C. RL Environment Description

The RL environment trains whole-body lifting on varied boxes, using defined episode outcomes, normalized observations, and a reduced continuous action parameterization.

1) Task Description:

  • Episodes target lifting objects from the ground, last at most 60 seconds, and run the policy at 20 Hz.The simulated training set contains 1,000 boxes sampled across parameter ranges chosen to fit the workspace while precluding simple end-effector grasping.Source: 1) Task Description · p. 6, Table I · p. 6
  • Success means lifting the box 0.5 m above its starting height; tipping is more than 80 degrees from vertical, and time-limit truncation counts as a slip.Boxes begin at randomized positions and orientations, including ±0.1 m translation along x and ±60° rotation about z.Source: 1) Task Description · p. 6

2) Observations:

  • The agent receives a normalized observation vector formed from measurements taken directly from MuJoCo and scaled to [-1,1] using specified bounds.Source: 2) Observations · p. 6

3) Actions:

  • The physical controls are 24 pressure commands and one elevator-height command, parameterized for learning by a smaller continuous action vector.The height command is bounded in meters, while differential pressure commands are converted into antagonistic pressures centered at 150 kPa.Source: 3) Actions · pp. 6–7
  • Policy actions are low-pass filtered, unnormalized to the physical bounds, mapped to pressure and height commands, and then sent to the robot.Source: 3) Actions · p. 7

4) Reward Engineering for Whole-Body Grasping:

Reward engineering combines sparse task outcomes with a motion-primitive term that guides exploration toward a manually designed approach-and-grasp trajectory. The authors report that this was more stable than multiple hand-shaped reward terms.

4) Reward Engineering for Whole-Body Grasping:

  • The sparse task reward assigns -2 when the box tips and 10 when its height exceeds the initial height by 0.5 m.Source: 4) Reward Engineering for Whole-Body… · p. 7
  • Reward shaping was difficult because the elevator moves more slowly than the arms, while high-dimensional exploration and reward hacking can produce undesirable local minima.Source: 4) Reward Engineering for Whole-Body… · p. 7
  • A single motion-primitive reward proved more stable and better at avoiding local minima than several rewards for stability, proximity, contact, and safety.Unlike behavior cloning or inverse RL, it guides exploration through reward shaping rather than learning directly from demonstrations.Source: 4) Reward Engineering for Whole-Body… · p. 7
  • The primitive uses two manually chosen differential-pressure waypoints, approach and grasp, joined by piecewise linear interpolation over an episode.The waypoints were selected in a few minutes of simulation experimentation and encode avoiding ground collisions, using both arms, and delaying grasp until the torso is close.Source: 4) Reward Engineering for Whole-Body… · p. 7, Figure 6 · p. 7
  • The shaped primitive term is a quadratic exponential comparing the policy’s predicted action with the primitive’s action at each timestep.The total reward combines this shaped term with sparse task-completion and failure rewards.Source: 4) Reward Engineering for Whole-Body… · p. 7

5) Learning Algorithm:

The learning setup combines PPO with an action-space motion primitive, then evaluates the simulation policy and transfers it to hardware. The experimental protocol proceeds from simulation tradeoffs to simulation evaluation and finally zero-shot hardware transfer.

5) Learning Algorithm:

  • PPO jointly trains an actor and critic using clipped policy, value-function, and entropy terms, with separate networks and two hidden layers.Clipping limits large policy updates, the value loss supports credit assignment, and the entropy bonus encourages exploration.Source: 5) Learning Algorithm · p. 7
  • The waypoint primitive moves through approach, grasp, and lift states, using a 50-step counter and elevator-height condition to advance.It constructs the motion-primitive reward term from elevator height and approach and grasp offsets.Source: 5) Learning Algorithm · p. 8

D. Experimental Protocols:

  • Experiments proceed from simulation speed–accuracy tests to policy evaluation in simulation and then zero-shot sim-to-real transfer.The PPO hyperparameters are reported in Table III and were identified empirically.Source: D. Experimental Protocols · p. 8, Table III · p. 8

1) Simulation Speed vs Accuracy:

The section evaluates simulation speed and kinematic accuracy through single-thread timing, CC–UJ pose comparisons, and a non-constant-curvature approximation test. These experiments examine performance across timestep and disk-count settings and compare model predictions under sampled configurations.

1) Simulation Speed vs Accuracy:

  • Single-thread simulation speed is measured across timesteps from 0.1 to 10 ms and disk counts from 2 to 64, averaging 10,000 steps per setting.Real-time factor is simulated time divided by wall-clock time; multithreading improves effective speed but has configuration-dependent returns.Source: 1) Simulation Speed vs Accuracy · p. 8
  • The CC–UJ experiment compares tip-pose predictions for disk counts from 2 to 64 over equally spaced CC configurations spanning −2.1 to 2.1 radians.For each configuration, inverse kinematics selects UJ angles that minimize the difference between CC and UJ predicted tip poses; residual error is treated as model mismatch.Source: 1) Simulation Speed vs Accuracy · p. 8
  • The pose objective combines tip-position and orientation error, while a regularizer encourages the optimizer to center the UJs.The orientation term uses a rotation-matrix logarithm and axis-angle representation, with λ balancing position and orientation.Source: 1) Simulation Speed vs Accuracy · pp. 8–9
  • The non-CC experiment tests whether coarser UJ models capture localized, non-constant-curvature deformation during full manipulation episodes.It treats a slow 64-disk simulation as ground truth because measuring true hardware continuum-joint shape is beyond the study’s scope.Source: 1) Simulation Speed vs Accuracy · p. 9

2) Policy Training:

The policy-training section contrasts guided learning with a shaped-reward baseline, then evaluates learned behavior across randomized objects and perturbations. The evaluation compares the learned policy with an open-loop primitive and quantifies reactive action changes.

2) Policy Training:

  • The shaped-reward baseline combines chest proximity, contact count, and box-lifting height, with its task reward matching the guided reward’s task term.The authors describe it as a natural baseline inspired by prior shaped-reward approaches to multi-step tasks.Source: 2) Policy Training · p. 9

3) Policy Evaluation in Simulation:

  • Policy evaluation uses 1,000 randomly generated boxes and examines outcomes and their rank correlations with experimental variables to assess generalization.The object-wise analysis records episode outcomes and uses Spearman correlation to identify outcome dependence on inputs.Source: 3) Policy Evaluation in Simulation · p. 9
  • The learned policy and open-loop primitive are compared on identical initial poses for 1,000 boxes, with outcomes organized in a transition matrix.Under perturbation, each method is also tested in 1,000 trials using alternating one-second applications of a 100 N downward force.Source: 3) Policy Evaluation in Simulation · p. 9
  • Corrective action is the difference between perturbed and unperturbed policy actions, normalized by the maximum 13-dimensional action norm.The analysis summarizes this measure over a 100-step, five-second window after perturbation onset and groups trials by outcome transitions.Source: 3) Policy Evaluation in Simulation · pp. 9–10

4) Zero-shot Sim-to-Real Transfer:

The section reports simulation accuracy and speed choices alongside the zero-shot hardware evaluation protocol. It describes the conditions used to transfer the policy and the model-error measurements used to assess whether the simulation and observations support learning.

4) Zero-shot Sim-to-Real Transfer:

  • The hardware transfer uses five boxes spanning 0.5–10 kg, with five trials per box and no hardware fine-tuning.The simulated boxes assume centroid-centered, uniform mass properties, unlike the real boxes weighted at the bottom; this discrepancy was not explicitly represented in training.Source: 4) Zero-shot Sim-to-Real Transfer · p. 10

A. Simulation Speed vs Accuracy:

  • With 5 disks and a 5 ms timestep, simulation runs at 65x single-threaded real time; parallelization on 16 threads reaches approximately 550x.Fewer than five disks made lifting difficult, while larger timesteps could cause numerical problems under many contact constraints.Source: A. Simulation Speed vs Accuracy · p. 10, Figure 7 · p. 10
  • Tip-pose errors between CC and UJ models depend on configuration and decrease toward zero as disk count increases, with larger bends producing larger errors.The comparison uses position and orientation error surfaces and box plots.Source: A. Simulation Speed vs Accuracy · p. 10, Figure 8 · p. 11
  • A 64-disk simulation treated as ground truth yields average tip-position error of 3 mm and orientation error of 0.0055 radians for the CC estimate.The authors deem this performance satisfactory for large-scale grasping and lifting and conclude that the configuration observation q is suitable for policy learning.Source: A. Simulation Speed vs Accuracy · pp. 10–11
  • The study identifies a speed–accuracy boundary: too few disks increase errors enough to make the sim-to-real gap large enough to prevent successful hardware deployment.Faster simulation favors larger timesteps and fewer disks, while accuracy improves with increased disk count.Source: A. Simulation Speed vs Accuracy · p. 11

B. Policy Training

Guiding reinforcement learning with a motion primitive produces more stable training than the shaped-reward comparison and improves outcomes, especially under perturbation. Simulation analyses show reactive corrections can recover grasps but can also overcorrect, while object properties have limited correlations with success.

B. Policy Training

  • With guided reward, success plateaued near 1.0 after 15 million steps; shaped reward peaked earlier and remained unstable with premature-grasp failures.The shaped-reward success rate peaked at 3E7 steps.Source: B. Policy Training · p. 11, Figure 10 · p. 12

C. Policy Evaluation in Simulation

  • Across 1000 randomly sampled boxes, the policy achieved 94.1% success, and object properties were not strongly correlated with success rate.Taller boxes were easier to lift (z-size ρ = .14), while heavier boxes had a slight negative success correlation (ρ = −.08).Source: C. Policy Evaluation in Simulation · pp. 12–13
  • On 1000 unperturbed test objects, learning increased overall successes from 931 to 941, though it also converted some primitive successes into failures.The Stuart-Maxwell test found a statistically significant effect (p = 1.4e-2).Source: Table IV · p. 13, C. Policy Evaluation in Simulation · p. 13
  • Under 100 N perturbation, the learned policy achieved 850 successes versus 822 for the primitive, reducing slips 55% but increasing tips 51%.The largest transition was Slip →Tip, occurring 48 times.Source: Table V · p. 13, C. Policy Evaluation in Simulation · p. 13
  • After perturbations, the policy modulated corrective actions; successful recoveries used mean actions of 0.40% for Slip →Success and 0.56% for Tip →Success.The open-loop primitive cannot respond reactively to external disturbances.Source: C. Policy Evaluation in Simulation · p. 14
  • The most common failure transition was Slip →Tip, with 48 instances, as large corrections sometimes tipped the box instead of recovering it.These failures had the highest mean effort (1.02%) and peak effort (33.1%).Source: C. Policy Evaluation in Simulation · p. 14

D. Zero-shot Sim-to-Real Transfer

The learned policy transferred to Baloo without fine-tuning and achieved an 88% hardware success rate, closely matching simulation. Smaller objects remained a practical challenge because arm self-collision could jam the grasp.

D. Zero-shot Sim-to-Real Transfer

  • Zero-shot transfer required no fine-tuning or pre-learning hardware validation; the authors attribute the transfer to qualitative manual calibration of masses, stiffness, and damping.Hardware slip rate was 12% versus 3.4% in simulation, while tip rates were 0% and 2.5%, respectively.Source: D. Zero-shot Sim-to-Real Transfer · p. 14
  • Smaller objects were most challenging: arm self-collision with a post-training protective cover could jam the grasp, and the cover was absent from simulation.The authors suggest this mismatch could explain the higher hardware slip rate.Source: D. Zero-shot Sim-to-Real Transfer · pp. 14–15
  • The policy achieved 88% success on hardware, close to the simulation rate of 94.1%, across 25 hardware trials.All five objects were successfully lifted at least once.Source: D. Zero-shot Sim-to-Real Transfer · p. 14, Figure 13 · p. 15

E. Emergent Whole-body Re-grasping Behavior

The learned policy re-grasps after objects are displaced, a behavior not encoded in its motion primitive or reward. The conclusion frames guided learning as producing reactive closed-loop behavior, while noting dependencies and failure modes that limit the current system.

E. Emergent Whole-body Re-grasping Behavior

  • Re-grasping emerged without being specified in the reward or motion primitive, which itself acts as an inflexible state machine.The authors attribute the reactive closed-loop controller to combining policy learning with a reward informed by an open-loop trajectory.Source: E. Emergent Whole-body Re-grasping… · p. 15
  • The policy re-approached and successfully lifted a box again after it was knocked from its grasp and pushed elsewhere.The same reattempt behavior occurred when another box slipped following self-collision; the authors suggest the policy may infer grasp slippage from an unexpected measured box position.Source: E. Emergent Whole-body Re-grasping… · p. 15, Figure 14 · p. 16

V. CONCLUSION

  • The conclusion reports that guided learning transforms the open-loop primitive into a reactive closed-loop policy capable of re-grasping.It presents re-grasping as an example of the learned policy's reactivity.Source: §V · p. 15

VI. LIMITATIONS AND FUTURE WORK

  • The method requires a hand-designed motion primitive for each new task, and reactive policies can over-correct forcefully under unknown disturbances.The authors identify learning primitives from a few non-expert demonstrations and training with perturbations as future directions.Source: §VI · p. 15
  • Hardware failures included self-jamming during initial approach; the authors also report overly tight grasps without force or tactile feedback.Proposed future work includes refining pre-grasp approach, integrating tactile sensing, and investigating higher-fidelity simulation and uncertain dynamics.Source: §VI · p. 16
  • The current system uses external motion tracking for object pose and robot proprioception; vision-based estimation remains future work.Source: §VI · p. 16

Related papers

Loading 2509.23556v1…