Source-linked AI summary
Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
Jan Matas, Stephen James, Andrew J. Davison
TL;DR
Deformable-object manipulation poses modeling and data challenges, and sim-to-real transfer had not been demonstrated for deformable policies. The paper combines deep RL improvements and domain randomisation to train cloth-manipulation policies entirely in simulation, then deploys them on three real-world tasks without real deformable-object training. The authors report successful transfer, while identifying grasping, motion, and simulation-support limitations.
Problem
Deformable manipulation is difficult because of large configuration spaces and challenging behavior modeling, while sim-to-real transfer of deformable policies had not been investigated.
Method
An improved DDPG agent, seeded with 20 demonstrations, is trained entirely in simulation with sparse rewards, auxiliary predictions, and domain randomisation.
Results
The agent learned three long-horizon cloth tasks and transferred the resulting policies to the real world without further training on real deformable objects.
Takeaways & Limitations
Deep RL with domain randomisation can be applied to cloth manipulation across folding and hanging tasks without explicit cloth-behavior modeling.
Takeaways & Limitations
Failures commonly involved grasping, overly fast or inaccurate motion, and excessive draping; camera-position sensitivity remained a transfer issue.
Abstract
from arXiv · showhide
We have seen much recent progress in rigid object manipulation, but interaction with deformable objects has notably lagged behind. Due to the large configuration space of deformable objects, solutions using traditional modelling approaches require significant engineering work. Perhaps then, bypassing the need for explicit modelling and instead learning the control in an end-to-end manner serves as a better approach? Despite the growing interest in the use of end-to-end robot learning approaches, only a small amount of work has focused on their applicability to deformable object manipulation. Moreover, due to the large amount of data needed to learn these end-to-end solutions, an emerging trend is to learn control policies in simulation and then transfer them over to the real world. To-date, no work has explored whether it is possible to learn and transfer deformable object policies. We believe that if sim-to-real methods are to be employed further, then it should be possible to learn to interact with a wide variety of objects, and not only rigid objects. In this work, we use a combination of state-of-the-art deep reinforcement learning algorithms to solve the problem of manipulating deformable objects (specifically cloth). We evaluate our approach on three tasks --- folding a towel up to a mark, folding a face towel diagonally, and draping a piece of cloth over a hanger. Our agents are fully trained in simulation with domain randomisation, and then successfully deployed in the real world without having seen any real deformable objects.
1 Introduction
Deformable-object manipulation remains challenging because of large configuration spaces and difficult behavior modeling, while sim-to-real transfer had not been studied for deformable policies. This work trains deep RL cloth-manipulation policies entirely in simulation and evaluates them on three real-world tasks.
- Deformable objects create large configuration spaces, difficult behavior modeling, and substantial configuration changes during manipulation.
- Existing deformable-object work largely targets 2D folding, while explicit cloth models are sensitive to deformation parameters and generalize poorly.
- The algorithm was evaluated on three tasks: folding a towel to a tape mark, hanging a towel on a hanger, and diagonally folding cloth.
- No prior study had investigated sim-to-real transfer for deformable-object policies using methods such as domain randomisation.
- The paper trains an improved DDPG agent purely in simulation, seeded with 20 demonstrations, using sparse task-completion rewards and domain randomisation.
2 Related Work
Related work combines conventional model-based and heuristic cloth manipulation with emerging learning-based approaches, while domain randomisation offers a route for simulation-to-real transfer. Prior sim-to-real demonstrations largely concern rigid-object tasks rather than deformable manipulation.
- Conventional cloth manipulation identifies configurations visually with hand-engineered heuristics and uses them to parameterize trajectories or mathematical cloth models.
- Deep learning has seen limited application to cloth manipulation despite broader use in rigid manipulation and other robotic control domains.
- Domain randomisation samples simulation parameters around noisy ground-truth estimates so policies learn to ignore minor environmental variations.
- Prior successful domain-randomisation transfer involved a supervised-learning pick-and-place agent moving a cube to a basket amid distractors and variable lighting.
3 Background
The method uses partially observed continuous-control RL with an actor operating on camera and proprioceptive observations and critics using privileged simulator state. Several DDPG extensions address sparse rewards, demonstrations, replay prioritization, and value overestimation.
- MDP formulation: The task is modeled as a partially observable MDP with full states, actions, transitions, rewards, and a discount factor; rewards are sparse and awarded only upon task completion.
- DDPG: DDPG learns continuous-action policies with an actor mapping observations to actions and a critic estimating action quality through Bellman loss minimization.
- DDPG: Noisy actions support exploration, while replay buffers reduce temporal correlations and target networks limit instability in recursive Q-value estimation.
- Replay and demonstrations: Prioritized replay samples transitions according to temporal-difference error while importance-sampling weights correct the resulting bias.
- DDPG extensions: N-step returns propagate rewards across multiple subsequent transitions, and combined one-step and N-step losses are weighted by separate hyperparameters.
- DDPG extensions: Demonstrations seed replay and behavioral cloning, reset-to-demonstration shortens sparse-reward subtasks, and TD3 uses twin critics with delayed target updates.
- Asymmetric actor-critic: The asymmetric actor-critic gives RGB partial observations to the actor and low-dimensional simulator state to the critic, reducing parameters and improving critic accuracy.
4 Method
The method adapts deep reinforcement learning for deformable-object manipulation by combining several DDPG improvements, auxiliary prediction, demonstration-based behavior cloning, and domain randomisation. Initial DDPG failed on all proposed environments, motivating the integrated learning algorithm.
- The simulation uses stochastic grasp failures and multiple fingertip anchors to expose unsuccessful grasps while avoiding unnatural cloth spinning.Grasps deliberately fail in 5% of cases, and anchors are placed at the middle and both extremities of each fingertip.
- Domain randomisation varies visual, geometric, robot, camera, lighting, and reflectance parameters around noisy ground-truth estimates to facilitate policy transfer.Textures use Perlin noise, while values are sampled from normal or uniform distributions.
- DDPG initially failed to solve any proposed environment, so the authors implemented multiple extensions to improve the learning algorithm.They did not use TD3 target Q-value regularisation because it harmed performance on these tasks.
- The actor loss combines the critic objective with behavior-cloning and auxiliary losses.The auxiliary outputs predict cloth corner positions, tape y-coordinate, and hanger y-coordinate; their loss is mean squared error.
- Demonstration transitions receive an additional priority boost proportional to the maximum losses in the current minibatch.The fixed demonstration-priority constant was replaced because TD-error magnitudes varied by multiple orders of magnitude across training epochs.
5 Experiments
The experiments evaluate three newly implemented deformable-object environments, simulation training with demonstrations and domain randomisation, ablations of DDPG improvements, and real-world transfer without additional training.
- 5.1 Cloth manipulation environments: Three environments—Tape, Hanging, and Diagonal Folding—provide RGB observations, low-dimensional state inputs, and sparse +100 success rewards for a 7DOF Kinova Mico arm.The 4-dimensional action controls end-effector velocities and gripping velocity.
- 5.2 Simulation results and ablation studies: Simulation training used 20 demonstrations and approximately 80k transitions with domain randomisation; final success rates averaged three random seeds.Each experiment took approximately 24 hours on one GeForce GTX TITAN.
- 5.2 Simulation results and ablation studies: The full algorithm either outperformed or matched ablated variants, while removing joint angles from the actor input harmed performance.This indicates that images alone did not provide sufficiently accurate gripper-position inference.
- 5.2 Simulation results and ablation studies: Twin Critic and pre-training offered questionable value because their apparent improvements traded against additional computation.Pre-training added a constant 7 minutes, while two critics increased runtime by 1%.
- 5.2 Simulation results and ablation studies: Auxiliary predictions, Behavioural Cloning, and demonstration prioritisation convincingly improved agent performance.Prioritisation makes demonstrations more likely to be sampled despite forming only a small portion of the replay buffer.
- 5.3 Sim-to-real transfer: Real-world evaluation used 30 trials per task without additional training, but grasping failures remained prominent, especially for thin face towels.Successful grasping allowed only roughly 1 cm of z-axis error, and the real cloth was stiffer than in simulation.
- 5.3 Sim-to-real transfer: Heavy texture randomisation worsened transfer, whereas camera randomisation was essential; the transferred agent remained sensitive to camera position.The authors attribute the texture result to difficulty identifying invariant orientation features.
6 Conclusion and Future work
The paper extends deep reinforcement learning from rigid to deformable cloth manipulation, training a task-agnostic agent entirely in simulation for three long-horizon tasks. It identifies simulator support for deformable objects as a primary research constraint and calls for accurate, shared benchmark environments.
- 6 Conclusion and Future work: The task-agnostic Deep RL algorithm bypasses explicit cloth modelling, uses sparse task-completion rewards, and learns three long-horizon cloth tasks from 20 demonstrations.Training occurred entirely in simulation with domain randomisation for policy transfer.
- 6 Conclusion and Future work: The learning algorithm incorporated nine literature-based improvements, with ablation studies examining their contributions.
- 6 Conclusion and Future work: The authors identify limited simulator support for deformable objects as the primary factor constraining further research.They propose accurate deformable-object grasping models and released benchmark environments as future directions.