Source-linked AI summary
Learning to Manipulate Deformable Objects without Demonstrations
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, Pieter Abbeel
TL;DR
The paper tackles sample-inefficient model-free visual RL for deformable-object manipulation without demonstrations. It uses conditional pick-place learning with random training picks and MVP-based test-time picking, achieving faster learning and real-robot cloth and rope transfer.
Problem
Model-free visual RL avoids explicit state estimation and object dynamics models for deformable objects, but its poor sample complexity limits manipulation learning without human demonstrations.
Method
The method learns a placing policy conditioned on random pick points and derives the picking policy by selecting the point with Maximum Value of Placing.
Results
The framework achieves an order of magnitude faster learning than independent action spaces on visual RGB deformable-manipulation tasks and transfers to real PR2 cloth and rope tasks.
Takeaways & Limitations
Conditional action structure and placing-value-informed picking improve learning for deformable-object manipulation and support sim-to-real transfer without additional real-world training or demonstrations.
Abstract
from arXiv · showhide
In this paper we tackle the problem of deformable object manipulation through model-free visual reinforcement learning (RL). In order to circumvent the sample inefficiency of RL, we propose two key ideas that accelerate learning. First, we propose an iterative pick-place action space that encodes the conditional relationship between picking and placing on deformable objects. The explicit structural encoding enables faster learning under complex object dynamics. Second, instead of jointly learning both the pick and the place locations, we only explicitly learn the placing policy conditioned on random pick points. Then, by selecting the pick point that has Maximal Value under Placing (MVP), we obtain our picking policy. This provides us with an informed picking policy during testing, while using only random pick points during training. Experimentally, this learning framework obtains an order of magnitude faster learning compared to independent action-spaces on our suite of deformable object manipulation tasks with visual RGB observations. Finally, using domain randomization, we transfer our policies to a real PR2 robot for challenging cloth and rope coverage tasks, and demonstrate significant improvements over standard RL techniques on average coverage.
I. INTRODUCTION
The paper addresses deformable-object manipulation, where noncanonical states and complex dynamics challenge traditional planning and visual RL remains sample-inefficient. It introduces conditional pick-place learning and MVP picking to accelerate learning without demonstrations.
- Deformable objects lack an obvious canonical state and exhibit complex, nonlinear dynamics, complicating representation and traditional task-and-motion planning.
- Model-free visual RL can learn directly from images without explicit object models, but poor sample complexity has limited learning without human demonstrations.
- The proposed iterative pick-place action space encodes the strong dependence of placing on the selected pick point.
- MVP selects the pick point maximizing the learned placing value, while training the placing policy with random pick points.
- The method reports faster learning on rope and cloth tasks and transfer to real-robot manipulation using domain randomization without additional demonstrations.
II. RELATED WORK
Prior work has addressed deformable manipulation through planning, local control, and reinforcement learning, but complex dynamics and inefficient learning limit broad visual-RL applicability. Several RL approaches also rely on expert demonstrations.
- Motion-planning methods combine deformable simulations with efficient planning, but high-dimensional configuration spaces and complex dynamics motivate high-level planners or demonstrations.
- Local model-based controllers can control an end effector toward goals, yet simplified dynamics may leave them trapped in local minima under more complex dynamics.
- Visual RL has solved several rigid-object manipulation tasks but has not achieved wide applicability to deformable objects because learning is inefficient with complex dynamics.
- Many prior deformable-object RL methods require expert demonstrations, while concurrent work replaces human demonstrations with simulated demonstrations using state information.
III. BACKGROUND
The paper frames RL as a continuous, partially observable MDP in which policies act from observations and seek to maximize discounted future rewards.
- The MDP contains continuous states and actions, partial observations, transition probabilities, rewards, a discount factor, and an initial-state distribution.
- At each timestep, the agent selects an action from its policy, receives a reward, and transitions according to the environment’s transition distribution.
- The RL objective is to maximize the expected sum of discounted rewards, using observations rather than states in the partially observable setting.
B. Off Policy Learning
Off-policy RL improves sample reuse by retaining past experiences, and the paper uses SAC because actor-critic learning suits continuous control while balancing exploration and return.
- Off-policy RL stores past experiences in a replay buffer, allowing sample reuse and generally improving practical sample efficiency over on-policy learning.
- Off-policy methods include model-based RL, actor-critic learning, and Q-learning, distinguished by whether they learn dynamics, policies and values, or values alone.
- The paper chooses actor-critic learning for its stated stability, data efficiency, and suitability for continuous control.
- SAC is the base algorithm, using entropy regularization to trade off policy entropy and expected return and encourage exploration.
- SAC learns a parameterized Q-function and policy, with a target Q-value network and adaptive entropy coefficient.
IV. APPROACH
The approach structures deformable-object manipulation as iterative pick-and-place actions, then learns placing conditioned on random picks and derives picking through Maximum Value under Placing (MVP).
- Pick-and-place action space: Iterative pick-and-place actions move a selected object point to a placement location while allowing repeated operations for deformable manipulation.For deformable objects, other points move according to object kinematics and dynamics, producing informative reward after each action.
- Conditional action spaces: A joint policy directly outputs pick and place locations but fails to capture the strong conditional dependence of placing on picking.The place location depends heavily on the selected pick point.
- Conditional action spaces: The factored action space feeds the picking policy output into the placing policy, while introducing additional action-factor credit-assignment challenges in reinforcement learning.Early collapse of the picking policy can restrict exploration of the placing policy.
- MVP learning: The two-stage scheme trains the placing policy with uniformly random picks, then uses its pick-conditioned value approximator to derive the picking policy.This separates placing-policy learning from picking-policy learning and preserves broad pick exploration during training.
- MVP learning: MVP selects the available image pick location that maximizes learned placing value, producing an informed picking policy without explicitly training one.The learned placing policy uses random picks during training, whereas MVP is used during testing.
V. EXPERIMENTAL EVALUATION
The experiments evaluate MVP across simulation suites and then demonstrate real-world deformable-object manipulation using the learned policies.
- Experimental evaluation: The evaluation analyzes MVP across simulations before demonstrating real-world deformable-object manipulation with the learned policies.The section covers both simulation analysis and real-world manipulation.
A. Cloth Manipulation in Simulation
The simulation study uses custom MuJoCo deformable-object environments, evaluates state- and image-based learning, and demonstrates MVP on rope and cloth tasks.
- Simulation environments: Custom DM Control environments use MuJoCo composite objects to simulate and render deformable cloths and ropes.The environments support consistent analysis because common RL benchmarks lack deformable-object tasks.
- Rope: The rope task aims to stretch a 25-joint rope horizontally straight at the table center using pick points and relative place movements.Larger relative movements are constrained because they produce unstable simulations.
- Rope: For 64×64 rope images, the reward encourages segmented rope pixels to occupy the center row with an exponential penalty for distance from that row.The rope is randomized at episode initialization through a random action over the first 50 timesteps.
- Cloth: Cloth-Simplified uses an 81-joint 9×9 grid, restricts picks to four corners, and rewards overlap with the target flattened configuration.Cloth removes the four-corner restriction and allows picking any cloth pixel under image observations or any composite particle under state observations.
- MVP demonstrations: MVP demonstrations show horizontal rope straightening and centering plus cloth spreading from multiple starting states in simulation.Displayed images are approximately five actions apart for rope and ten actions apart for cloth.
B. Learning Methods for Comparison
The comparison includes random, independent, conditional, learned-placing-with-uniform-pick, and MVP policies, with MVP deriving picks from learned placing values.
- Compared methods: The study compares random, independent, conditional, learned placing with uniform pick, and MVP learning methods.These methods are evaluated across the rope and cloth environments.
- Baselines: The random baseline samples pick locations and robot place actions uniformly.
- Baselines: The independent baseline simultaneously outputs pick and place actions through a joint factorization.It is also labeled Independent to distinguish it from the Conditional baseline.
- Baselines: The conditional baseline models place selection given a previously selected pick location.Its policy factorization is p(apick|o) × p(aplace|apick, o).
- Proposed components: Learned placing with uniform pick trains p(aplace|apick, o) while sampling pick locations uniformly.
- Proposed components: MVP uses the learned placing-with-uniform-pick policy and chooses the pick by maximizing the learned Q-function.Independent and Conditional baselines receive an extra positive reward for selecting a valid object pick location.
C. Training Details
The experiments use SAC-based visual and state policies, with specified network architectures and task horizons. Real-robot demonstrations use MVP with domain-randomized simulation for rope and cloth spreading.
- Training setup: Simulation training uses SAC as the off-policy reinforcement-learning algorithm.The implementation modifies the rlpyt code-base.
- Network architectures: State-based policies use a two-layer 256-unit MLP, while image-based policies use a three-layer CNN followed by two 256-unit fully connected layers.The state and image models have approximately 150k and 200k parameters, respectively.
- Real-robot setup: MVP policies with domain-randomized simulation demonstrate real-robot manipulation for rope and cloth spreading tasks.Rope frames represent one pick-place action; each cloth frame represents 10 robot actions.
- Network architectures: Pick information is repeated 50 times before concatenation with state observations or image embeddings so both inputs are weighted equally.The authors report that this improves performance.
- Task configuration: The Rope horizon is 200 timesteps, while both Cloth environments use 120 timesteps.
D. Does conditional pick-place learning help?
Conditional pick-place learning improves learning speed and reward in challenging deformable-object tasks, especially with image observations. MVP further improves performance, while domain randomization preserves simulation performance during training.
- Conditional learning: 5–10X higher reward is achieved by the proposed method than baseline methods in image-based experiments.The baseline methods perform no better than random, while independent and conditional factored cloth policies can perform worse than random.
- Conditional learning: 2X faster learning is reported for the harder state-based cloth environment.The method also produces significantly higher rewards in the Cloth-Simplified environment.
- MVP: MVP consistently improves performance across state-based and image-based experiments by using placing values to inform picking without explicitly training a picking policy.The boost can be smaller than that from learned placing with uniform pick, which already achieves a high task success rate.
- Domain randomization: Domain randomization changes visual parameters such as lighting and textures and physical parameters such as cloth mass and joint friction.Simulation evaluation reports no performance degradation from domain randomization during MVP training.
- Real-robot transfer: The real-robot policy receives RGB images from a head-mounted PR2 camera and maps image-space pick and place points to robot actions.The mapping is fitted from 4–5 correspondences between robot coordinates and image pixels.
G. Evaluation on the real robot
MVP-trained policies successfully perform rope-spreading and cloth-spreading on a PR2 robot. The evaluation compares them with baselines using spread coverage, while the conclusion places the approach within broader deformable-manipulation research.
- Real-robot performance: MVP-trained policies successfully complete both rope-spreading and cloth-spreading tasks on the PR2 robot.A single domain-randomized policy can spread cloths of different colors.
- Evaluation protocol: The evaluation uses four start configurations for each task and the spread coverage metric.Rope policies run for 20 steps, whereas cloth policies run for 150 steps.
- Baseline comparison: The large gap between MVP-trained and independent policies supports the hypothesis that conditional structure is crucial for deformable-object manipulation.
- Conclusion: The paper presents conditional learning as significantly improving sample complexity and as a first from-scratch RL demonstration for deformable-object manipulation on a real robot.
- Future directions: The approach assumes an actor-critic algorithm and may be combined with learning-from-demonstration techniques for further performance improvements.