Source-linked AI summary
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
Xingyu Lin, Yufei Wang, Jake Olkin, David Held
TL;DR
Deformable-object manipulation lacks standardized RL benchmarks and remains difficult because of high-dimensional dynamics and partial observability. SoftGym introduces open-source simulated benchmarks with standardized interfaces and evaluates policies under multiple observation assumptions. The results show that visual-observation methods perform far below ground-truth-state methods on many tasks, highlighting substantial challenges for current RL algorithms.
Problem
Deformable-object manipulation lacks a common benchmark for comparing learning approaches, while its high-dimensional dynamics and observations challenge existing RL assumptions.
Method
SoftGym provides open-source simulated deformable-object benchmarks with standard Gym and Python interfaces, and evaluates representative policies under different observation spaces.
Results
Visual-observation reinforcement learning performs far below optimal performance on many tasks, whereas the Dynamics Oracle solves most tasks and serves as an upper bound.
Takeaways & Limitations
SoftGym standardizes evaluation and exposes challenges that can guide future algorithmic development for deformable-object manipulation.
Takeaways & Limitations
SoftGym-Robot was planned for release only after obtaining permission from Nvidia, and its baselines use ground-truth rewards unavailable outside simulation.
Abstract
from arXiv · showhide
Manipulating deformable objects has long been a challenge in robotics due to its high dimensional state representation and complex dynamics. Recent success in deep reinforcement learning provides a promising direction for learning to manipulate deformable objects with data driven methods. However, existing reinforcement learning benchmarks only cover tasks with direct state observability and simple low-dimensional dynamics or with relatively simple image-based environments, such as those with rigid objects. In this paper, we present SoftGym, a set of open-source simulated benchmarks for manipulating deformable objects, with a standard OpenAI Gym API and a Python interface for creating new environments. Our benchmark will enable reproducible research in this important area. Further, we evaluate a variety of algorithms on these tasks and highlight challenges for reinforcement learning algorithms, including dealing with a state representation that has a high intrinsic dimensionality and is partially observable. The experiments and analysis indicate the strengths and limitations of existing methods in the context of deformable object manipulation that can help point the way forward for future methods development. Code and videos of the learned policies can be found on our project website.
1 Introduction
Deformable-object manipulation is difficult because of high-dimensional states and complex dynamics, while existing RL benchmarks do not support direct comparison across approaches. SoftGym addresses this gap with standardized simulated environments and evaluates how observation choices affect performance.
- Deformable-object manipulation is challenging because robots face high-dimensional state representations and complex dynamics.
- Existing learning-based studies use different task variants, simulators, and robot setups, preventing direct comparison and leaving no common benchmark.
- Traditional RL benchmarks commonly assume directly observed, low-dimensional states that are difficult to perceive or define for deformable objects.
- SoftGym provides open-source simulated benchmarks with a standard OpenAI Gym API and Python interface for new environments.
- Visual-observation learning performs much worse than ground-truth-state learning on many deformable-object manipulation tasks.
2 Related Works
Prior deformable-object manipulation research combines traditional planning and newer learning-based approaches, but existing RL benchmarks generally omit deformable objects and use low-dimensional state representations.
- Traditional approaches detect visual features and combine motion planning with analytical deformable-object models.
- Analytical planning approaches often struggle with the large configuration spaces created by deformable objects’ many degrees of freedom.
- Recent learning-based work explores image-based cloth policies and towel-spreading pick-and-place policies, often training in simulation before robot transfer.
- Established RL benchmarks omit deformable objects and typically represent the full system state with a low-dimensional vector.
3 Background: Deformable Object Modeling in FleX
SoftGym models deformable objects in Nvidia FleX as particle systems governed by physical constraints, with task visualizations spanning rope, cloth, and fluid environments.
- FleX represents each deformable object with particles and internal constraints in a particle- and position-based dynamical system.
- Each particle has position, velocity, and inverse mass attributes, while constraints depend on the positions of relevant particles.
- FleX integrates velocities to predict particle positions, then projects those positions onto the feasible set defined by constraints.
- Fluids use constant-density constraints based on each particle and its neighbors to enforce incompressibility.
- Rope, cloth, and self-collision are modeled through springs and stretching, bending, and collision constraints.
4 SoftGym
SoftGym organizes deformable-object manipulation tasks across benchmark tiers and action abstractions, covering rope, cloth, and fluids while exposing high-dimensional dynamics and varied task objectives.
- SoftGym’s underlying dynamical state ranges from hundreds to thousands of dimensions because objects are represented by particles.
- SoftGym-Medium has six tasks, SoftGym-Hard adds four more challenging tasks, and SoftGym-Robot uses Sawyer or Franka robot action spaces.
- 4.1 Action Space: Abstract picker and cup actions decouple high-level planning from low-level grasping, while SoftGym-Robot provides Cartesian end-effector control.
- 4.1 Action Space: SoftGym-Robot was planned for release only after obtaining permission from Nvidia.
- 4.2 Tasks: TransportWater and PourWater evaluate moving or rotating cups while limiting spilling or maximizing water delivered to a target.
- 4.2 Tasks: Rope and cloth tasks evaluate straightening, spreading, folding, dropping, and configuration matching from varied initial states.
- On an Nvidia 2080Ti, SoftGym runs about 4x faster than real time with rendering, while one million simulation steps take 6 hours.
5 Methods Evaluated
SoftGym evaluates representative policy-search methods under different assumptions about dynamics and deformable-object state access, including high-dimensional and visual observations. The benchmark uses ground-truth rewards while comparing oracle, reduced-state, and observation-driven approaches.
- 5 Methods Evaluated: SoftGym groups representative policy-search algorithms by their assumptions about underlying dynamics and particle positions.The baselines support analysis of distinct challenges in deformable-object manipulation.
- 5 Methods Evaluated: The Dynamics Oracle uses ground-truth particle positions, velocities, and dynamics with gradient-free action-sequence optimization.CEM searches for the action sequence with the highest return under the ground-truth dynamics model.
- 5 Methods Evaluated: The Full State Oracle supplies ground-truth particle positions and robot or picker proprioception to an SAC policy without ground-truth dynamics.It uses a standard multi-layer perceptron as the policy architecture.
- 5 Methods Evaluated: The Reduced State Oracle uses a hand-defined, perfectly estimated subset of the full state with SAC, avoiding direct learning from high-dimensional state spaces.For cloth, the reduced state consists of the four corner positions; other tasks use task-specific keypoints or object variables.
- 5 Methods Evaluated: The benchmark also evaluates algorithms operating directly on high-dimensional observations because low-dimensional state representations may not be accurately inferable.CURL-SAC, DrQ, and PlaNet represent image-based, augmentation-based, and latent-dynamics approaches evaluated in the benchmark.
6 Experiments
SoftGym experiments compare reinforcement-learning methods across observation assumptions and evaluate whether deformable-object tasks expose challenges absent from simpler benchmarks. Results show that image-based learning and full-state policies struggle, while future prediction failures reveal difficulties modeling deformable dynamics.
- Experimental setup: Experiments compare current reinforcement-learning algorithms across full-state, reduced-state, and image-based observation spaces.Each task uses normalized performance, five random seeds, environment variations split into training and evaluation sets, and algorithm-specific training horizons.
- Benchmarking results: The Dynamics Oracle performs best on most tasks and serves as an upper performance bound because real-world systems usually lack particle states and dynamics.It uses ground-truth particle information and dynamics unavailable in real-world manipulation.
- Benchmarking results: Reduced-state policies perform well when task-relevant information is captured but poorly when representations omit configuration details, as in SpreadCloth.The four cloth-corner positions are insufficient to reason about the cloth configuration in SpreadCloth.
- Benchmarking results: Image-based methods perform far below optimal performance on many tasks, especially StraightenRope, SpreadCloth, and FoldCloth, and lag reduced-state policies on some tasks.The learning curves suggest that additional training may not substantially improve performance on the especially difficult tasks.
- Benchmarking results: The Full State Oracle performs poorly on all tasks, demonstrating difficulty learning with variable-size, high-dimensional deformable-object states.Its input includes the positions of all object particles.
- Difficult future prediction: PlaNet fails to predict spilled water and cloth shape in open-loop future observations, indicating that deformable-object visual dynamics are difficult to learn.Figure 3 compares predicted frames after an initial five-frame history with ground-truth future observations.
7 Conclusion
SoftGym concludes that deformable-object manipulation requires benchmarks spanning high-dimensional states, partial observability, and varied physical objects. Its environments expose current reinforcement-learning challenges while standardizing observations and actions for comparison and future development.
- 7 Conclusion: SoftGym benchmarks manipulation of rope, cloth, and fluids whose underlying dynamic state ranges from hundreds to thousands of dimensions.The benchmark is intended to advance reinforcement learning in complex environments with intrinsically high-dimensional states.
- A.1 Observation Space: Each task provides full-particle, reduced-state, and image-based observation spaces, enabling comparisons across state representations.Image observations are RGB images rendered at 128×128×3 in the reported experiments.
- A.1 Observation Space: Reduced states are task-specific: water tasks encode cup and water properties, rope tasks use ten keypoints, and cloth tasks use four corners.Picker or robot positions are also included where applicable.
- Action Space: All environments normalize agent actions to [−1, 1], while underlying controls vary from one-dimensional cup motion to picker or cup pose changes.Picker actions include position increments and a binary-like picking-state control.
A.3 Task Variations
SoftGym varies deformable-object tasks across object properties, initial configurations, geometry, and task-specific quantities to reflect practical manipulation variability.
- Illustration: SoftGym-Hard task variations are omitted from the supplementary illustration because they are similar to the displayed task variations.Each image depicts the task after the initial reset.
- Variation design: Task variations cover object size, shape, physical properties, initial configurations, and task-specific spatial or volume parameters.The benchmark summarizes these variations across tasks in Table 1.
- Water manipulation: PourWater and related tasks vary cup dimensions, cup separation, and water volume, with particle-based geometry determining container dimensions.PourWaterAmount additionally samples the goal volume.
- Cloth manipulation: SpreadCloth and FoldClothCrumpled vary cloth dimensions and crumpled initial shapes, while FoldCloth varies cloth size from a flattened centered state.Cloth width and length are sampled from randint(60, 120) where applicable.
- Other deformable objects: DropCloth and DropFoldCloth vary cloth size, whereas StraightenRope varies the rope’s initial twisted shape through repeated particle perturbations.StraightenRope keeps rope length fixed.
A.4 Training and Evaluation
Training and evaluation use pre-sampled task variations, normalized performance bounds, and median results across multiple random seeds.
- Task splits: Each environment uses 1000 pre-sampled task variations, split into 800 training variations and 200 evaluation variations.The variations and their initial states are pre-computed for computational efficiency.
- Performance metric: Performance is normalized to [0, 1] using task-specific lower and upper bounds, although policies below the do-nothing lower bound can score below 0.The lower bound comes from a policy that always does nothing.
- Performance metric: The reported metric is normalized performance at the last time step unless explicitly specified otherwise.The performance metric is the reward without scaling.
B Algorithm Details
The benchmark reports action repetition and task horizons as task-level algorithm settings, with the values summarized in Table 2.
- Common settings: All tasks and algorithms use a discounting factor of γ = 0.99 when discounting applies.Action repetition and task horizon are separately summarized in Table 2.
- Task-level settings: Table 2 summarizes the action repetition and task horizon used for each task.The supplied passage does not enumerate the task-specific values.
B.1 CEM with Dynamics Oracle
CEM uses model predictive control with task-specific planning horizons, while the benchmark also specifies neural architectures and training procedures for image-based baselines. Planning-horizon sensitivity is task dependent.
- CEM: CEM performs 10 optimization iterations using model predictive control and 21K environment steps per decision.The number of candidate trajectories equals 21K divided by the planning horizon, and the top 10% are used as elites.
- CEM: CEM uses task-specific planning horizons summarized in Table 3.The table provides the horizon selected for each environment.
- CURL-SAC: CURL-SAC uses two-layer 1024-neuron MLPs for both policy and Q-value networks, with task-dependent reward scaling and learning-rate tuning.The implementation uses ReLU activations.
- PlaNet: PlaNet encodes images with a CNN, uses a 1024-dimensional embedding, and models dynamics with a 200-hidden-node GRU and two-layer 200-node MLPs.Latent overshooting is excluded because it provides little improvement over the one-step case.
- PlaNet: PlaNet training collects 900 experience steps per epoch and performs 100 gradient updates after an initial 5-episode replay-buffer warm-up.Training PlaNet for 1M steps takes around 120 hours on an Nvidia 2080Ti with 4 virtual CPUs and 40G RAM.
- Cloth baseline: The cloth-specific comparison uses a model-free image-based pick-and-place policy with heuristic cloth segmentation and 20-step waits before dropping.The official implementation and default hyper-parameters are used.
- Planning-horizon analysis: Performance is sensitive to CEM’s planning horizon in TransportWater, FoldCloth, and DropCloth, but relatively stable on the other tasks.The black bar denotes the performance reported in the main paper.