Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning

Andy Zeng; Shuran Song; Stefan Welker; Johnny Lee; Alberto Rodriguez; Thomas Funkhouser

Source-linked AI summary

Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning

Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, Thomas Funkhouser

arXiv:1803.09956v3Mar 27, 2018cs.ROcs.AIcs.CVcs.LGstat.ML

TL;DR

The paper asks how pushing and grasping can be learned as mutually supportive actions rather than planned separately. It jointly trains fully convolutional Q-learning policies from visual experience and reports complex, efficient picking behaviors in simulation and on a real robot, including generalization to novel objects.

Problem
Pushing and grasping are typically studied in isolation, leaving their synergistic sequencing relatively unexplored and difficult to supervise without hand-coded goals.
Method
The method jointly trains fully convolutional pixel-wise action-value networks for pushing and grasping using model-free, self-supervised Q-learning.
Results
The system learns complex pushing-and-grasping sequences in simulation and on a real robot, improves picking efficiency, and generalizes to novel objects.
Takeaways & Limitations
Pushing can be learned for its future grasping benefit, enabling mutually supportive manipulation in cluttered picking scenarios.

Abstract

from arXiv · show

Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors amid challenging cases of clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after only a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.edu

I. INTRODUCTION

The paper addresses the largely separate treatment of pushing and grasping by learning complementary policies jointly from visual experience. Its model-free approach uses self-supervised Q-learning to discover pushing behaviors that improve future grasping and supports complex picking in clutter.

Motivation: Pushing and grasping have complementary roles, with pushing rearranging clutter for grasping and grasping displacing objects for more precise pushing.These synergies motivate sequential manipulation that combines both action types.
Problem: Push and grasp planning have mostly been studied in isolation, while sequencing them remains relatively unexplored and difficult to reward without predefined goals.Pushing objectives such as separating objects or breaking clusters are difficult to define for model-based and data-driven methods.
Approach: The proposed system learns pushing and grasping jointly through self-supervised trial and error using model-free deep reinforcement learning and Q-learning.Pushing receives useful supervision when it eventually enables grasping, rather than from a hand-coded pushing objective.
Approach: Fully convolutional action-value networks map visual observations to dense pixel-wise expected rewards for pushing and grasping motion primitives.The pixel-wise formulation associates image pixels and orientations with end-effector actions at corresponding 3D scene locations.
Findings: Joint pushing and grasping policies enlarge the scenarios in which grasping succeeds and produce complex interactions that support more efficient picking.Reported behaviors include separating objects, pushing multiple blocks, and breaking clusters through reaction chains.
Evaluation: The approach is evaluated in simulated and real settings and is reported to generalize to novel objects.The paper provides experiments, ablations, qualitative videos, code, pretrained models, and simulation environments.

II. RELATED WORK

Prior work studies pushing, grasping, and their combination through increasingly data-driven methods, but often relies on object models, handcrafted goals, separated perception and control, or restricted scenarios. The paper positions its formulation as end-to-end, model-free joint learning for cluttered manipulation.

Non-prehensile manipulation: Classical pushing methods model contact dynamics, but friction variability and non-uniform surface friction can make their predictions inaccurate in real settings.Recent data-driven work often focuses on stable pushes for one object at a time, leaving larger-scale clutter consequences difficult to model.
Grasping: Model-based grasping commonly uses contact or mobility reasoning and retrieves grasps from databases of known 3D object models.More recent deep methods detect grasps from learned visual features without explicit object-specific knowledge.
Pushing with grasping: Combined push-grasp approaches include hand-coded push-grasping frameworks and model-free pushing toward target positions for fixed grasping algorithms.These methods use domain knowledge or concrete target goals to supervise pushing behaviors.
Pushing with grasping: Defining pushing goals is less clear when the grasping policy is data-driven and continuously learns its optimal behavior from experience.This motivates learning synergies from the future success of grasping rather than specifying pushing targets in advance.
This work: The proposed system instead uses jointly trained fully convolutional networks that represent pushing and grasping as dense pixel-wise Q-value maps.An RGB-D heightmap is processed for multiple rotations, with separate networks for the two motion primitives.
Pushing with grasping: A related reinforcement-learning pipeline selects among push and grasp proposals using hand-crafted features, object segmentation, model-based simulation, and assumptions about object shape.It was demonstrated mainly for convex objects in one two-object scenario.

III. PROBLEM FORMULATION

The task is formulated as a Markov decision process in which a robot selects manipulation actions, receives rewards, and learns a policy maximizing discounted future returns through Q-learning.

Each state-action transition produces a new state and an immediate reward under the robot’s policy.
The objective is to maximize the γ-discounted sum of future rewards over an infinite horizon.
Off-policy Q-learning trains a greedy deterministic policy that selects actions by maximizing their action-value estimates.
Learning minimizes the temporal-difference error between the estimated action value and a fixed target value.
The target value considers the best action among all available actions in the next state.

A. State Representations

States are represented as RGB-D heightmaps, while actions pair a pushing or grasping primitive with a 3D location projected from a heightmap pixel. The primitives discretize directions or orientations for executable robot motions.

State Representations: Each state is an RGB-D heightmap built by projecting fixed-camera data into a 3D point cloud and back-projecting it along gravity.The representation contains RGB color and height-from-bottom depth channels.
Primitive Actions: An action combines a pushing or grasping primitive with a 3D location projected from a state-representation pixel.
Primitive Actions: Grasping uses a top-down parallel-jaw motion at one of 16 orientations, with fingers descending 3cm below the selected location before closing.Stable, collision-free inverse-kinematics solves automatically plan arm motion for both primitives.

C. Learning Fully Convolutional Action-Value Functions

The method represents pushing and grasping with separate fully convolutional Q-functions that produce dense spatial action values from rotated RGB-D heightmaps. Shared convolutional structure exploits locality and geometric equivariance, while the reward scheme distinguishes successful grasps from environment-changing pushes.

Learning Fully Convolutional Action-Value Functions: Two fully convolutional networks separately estimate pushing and grasping Q-values from the heightmap state.Each network outputs a dense map whose pixel values represent expected future reward for a primitive at the corresponding 3D location.
Learning Fully Convolutional Action-Value Functions: Both networks use parallel DenseNet towers for RGB and depth inputs, followed by feature concatenation, convolutional layers, normalization, and upsampling.
Learning Fully Convolutional Action-Value Functions: The input heightmap is rotated into 16 orientations, producing 32 output maps for directional pushes and oriented grasps.Rotations reduce oriented primitives to horizontal actions in transformed inputs.
Learning Fully Convolutional Action-Value Functions: The selected action is the pixel and primitive with the largest predicted Q-value across the pushing and grasping networks.
Learning Fully Convolutional Action-Value Functions: Pixel-wise state and action parameterization gives predictions spatial locality and allows convolutional features to be shared across locations and orientations.This parameterization can reduce training-data requirements through translation and rotation equivariance.
Learning Fully Convolutional Action-Value Functions: Double Q-learning and duelling-network extensions are identified as possible improvements but are outside the paper’s focus.
Learning Fully Convolutional Action-Value Functions: Successful grasps receive reward 1, while pushes receive reward 0.5 when heightmap differences exceed a detection threshold.The push reward is intrinsic and is based on detectable environmental change.

E. Training details.

The networks are trained with a Huber-loss Q-learning procedure using fixed optimization settings, target-network updates, prioritized replay, and ε-greedy exploration. Self-supervised trial-and-error runs repeatedly clear randomly populated workspaces in simulation and the real world.

Training details: The Q-learning networks are trained with a Huber loss against target-network values, while target parameters remain fixed between updates.
Training details: Gradients are propagated only through the pixel and network corresponding to the executed action; all other pixels receive zero loss.
Training details: Optimization uses stochastic gradient descent with momentum, prioritized experience replay, and ε-greedy exploration annealed from 0.5 to 0.1.The fixed learning rate is 10^-4, momentum is 0.9, and weight decay is 2^-5.
Training details: Self-supervised training repeatedly drops randomly selected objects into a 0.4482m workspace and collects trial-and-error data until the workspace is empty.Simulation uses n = 10 objects, whereas real-world training uses n = 30.

F. Testing details.

Testing resets network weights for each run and stops after completion or more than 10 unchanged actions. The evaluation compares VPG with reactive grasping and pushing baselines in increasingly cluttered scenarios.

Testing procedure: Greedy deterministic policies can repeatedly execute the same action when the state and value estimates remain unchanged.VPG addresses this during testing with continued low-rate gradient updates after each action.
Testing procedure: Network weights are reset before each experiment run, which ends after all objects are grasped or more than 10 consecutive actions cause no environmental change.
Compared policies: VPG is evaluated against grasping-only and reactive pushing-and-grasping baselines using the same pixel-wise state and action-space formulation.The grasping-only baseline uses a single FCN, while P+G Reactive adds a separate pushing FCN trained with change-detection supervision.
Compared policies: Both baselines greedily select actions from affordances computed only from the current state rather than planning long-horizon strategies.
Evaluation settings: Policies are trained with random 10-object arrangements and evaluated on 10-object, 30-object, and manually engineered challenging clutter.

B. Evaluation Metrics

The evaluation uses novel object arrangements in simulated and real-world picking tests. Performance is measured by task completion, grasp success, and action efficiency.

Test setup: Tests require the system to pick and remove objects from a table under novel object arrangements.
Metrics: Completion rate measures finishing the task without more than 10 consecutive failed attempts, averaged across n approximately 10 or 30 runs.
Metrics: Average grasp success rate is computed per completion to measure grasping performance.
Metrics: Action efficiency is defined as the number of objects divided by the number of actions before completion, measuring how succinctly the policy finishes.
Test setup: Experiments cover both simulated and real-world platforms with objects placed in random and challenging arrangements.Simulation provides controlled comparisons and ablation studies, while the main objective is effective deployment on a real robot.

C. Simulation Experiments

Simulation studies evaluate VPG against reactive baselines and ablations in random and challenging clutter. Joint planning improves completion and grasping, while depth cues and long-term lookahead affect final performance.

Simulation setup: The simulation uses a UR5 arm, RG2 gripper, V-REP, Bullet Physics, and randomly selected shapes and colors from nine toy blocks.
Baseline comparison: In challenging arrangements, grasping-only achieves 0% completion in 5 of 11 test cases and 50-60% average grasp success when it completes.
Baseline comparison: Adding reactive pushing removes 0% completion cases and raises completion and grasp success, but lowers action efficiency because pushes are not supervised by grasping benefit.
VPG results: VPG achieves 100% completion in 5 of 11 challenging test cases, with higher completion, grasp success, and action efficiency than reactive alternatives.Its joint planning makes pushes more succinct in how they help grasping.
Ablations: VPG-noreward reaches 70-80% grasp success and learns pushes that increasingly help future grasps despite receiving no intrinsic pushing reward.
Ablations: Removing ImageNet pre-training does not substantially affect simulated sample efficiency or final performance, whereas removing depth lowers final grasping performance by about 15%.The results attribute the depth-channel effect to the importance of geometric height-from-bottom cues.
Ablations: With γ = 0.2, VPG-myopic improves early grasping faster but ultimately performs worse across most hard test cases in grasp success and action efficiency.The comparison supports a role for long-term planning in sequential manipulation stability and efficiency.

D. Real-World Experiments

Real-world experiments evaluate VPG on cluttered arrangements of toy blocks and novel objects. VPG learns effective pushing and grasping policies with sample-efficient training, later surpassing grasping-only performance while generalizing mainly to similarly shaped novel objects.

Experimental setup: VPG was evaluated on a UR5 robot with an RG2 gripper using 30+ toy blocks and separate novel-object scenes.The experiments used tabletop RGB-D perception and included generalization tests beyond the training objects.
Training behavior: VPG and grasping-only showed similar early performance improvement, despite VPG executing either a push or grasp per training step.The authors attribute this to early optimization of pushes that make subsequent grasps easier.
Training behavior: VPG later surpassed grasping-only in grasping performance and was less erratic in highly cluttered configurations.The authors associate this with avoiding long sequences of failed grasps.
Sample efficiency: Less than 2000 transitions produced effective pushing and grasping policies, corresponding to about 5.5 hours of real-robot training.The paper contrasts this training requirement with prior manipulation work using 10 million sample transitions.
Challenging arrangements: VPG performed synergistic pushing and grasping behaviors that completed cluttered picking tasks more efficiently than grasping-only policies.Video recordings document these behaviors in challenging real-world arrangements.
Novel objects: VPG generalized to novel objects with similar shape distributions but struggled with completely new shapes and reflective objects lacking depth data.The robot could still plan complex pushes that decluttered scenes containing novel objects.

VI. DISCUSSION AND FUTURE WORK

The paper presents a mutually supportive framework that learns pushing and grasping synergies from experience and executes complex sequences on a real robot. It also identifies limits in motion expressiveness, object diversity, and the range of manipulation primitives studied.

Discussion: The framework learns mutually supportive pushing and grasping policies from experience using pixel-wise deep reinforcement learning and affordance-based manipulation.The authors report that the policies learn complex real-robot action sequences within tractable training times.
Future work: The regular-grid heightmap parameterization improves learning efficiency but limits motion expressiveness, excluding more dynamic and parallel action combinations.The authors suggest richer motion parameterizations and varied robot contact surfaces as future directions.
Future work: Training uses blocks and tests only a limited range of other shapes, leaving broader generalization insufficiently evaluated.The paper proposes training on larger shape varieties.
Future work: The study addresses pushing and grasping but does not investigate other primitive manipulation actions such as rolling, toppling, squeezing, levering, or stacking.The authors identify expanding beyond these two primitives as a future direction.