Robotic Grasping using Deep Reinforcement Learning

• Off-policy reinforcement learning is presented as reducing training time and computation while learning a grasp policy from a small object set.Source: §I · p. 1
• The framework uses Grasp-Q-Network with multiple cameras and motor inputs to learn robust grasps.Source: §I · p. 1
• Continuous visual feedback drives a visual servoing mechanism for producing successful grasps.Source: §I · p. 1
• The double deep Q-learning network outperforms the proposed Deep Q-learning and vanilla Q-learning frameworks in maximizing grasp success.Source: §I · p. 2
• The work evaluates performance across different objects using single-view and multi-view camera setups, in simulation and on a real robot.Source: §I · pp. 1–2

• Grasping methods are grouped into analytical and empirical or data-driven approaches.Source: §II · p. 2
• RGB-D depth information can map 3D real-world coordinates into useful 2D-frame information for grasp detection and recognition.Source: §II · p. 2
• Vision-based deep-learning work commonly uses bounding boxes, while selecting the most successful or stable grasp remains important; sliding-window CNN detection can be slow,.Source: §II · p. 2
• Prior RL work centers on trial-and-error and delayed rewards; cited deep-Q systems address manipulation and grasping with pushing.Source: §II · p. 2
• Deep networks are presented as addressing memory, sample, and computational complexity in reinforcement learning; cited examples include PI2 and deterministic policy gradients.Source: §II · p. 2

• Grasping is modeled as a Markov Decision Process in which the robot observes the environment, acts under a policy, and receives reward, with the goal of maximizing discounted future returns.Source: §III · p. 2
• Observations combine an overhead RGB-D image, a wrist RGB image, and the robot arm’s joint angles.Source: §III · pp. 2–3
• The network processes visual and motor features and outputs grasp-success probabilities using a softmax function.Source: §III · p. 3

IV. PROPOSED APPROACH

• Overhead- and wrist-camera observations plus motor positions feed the Grasp-Q-Network, whose Q-values guide ε-greedy action selection.The network outputs grasp-success probabilities as Q-values for possible actions.Source: §IV · p. 3

A. Overview

• Continuous visual servoing adjusts motor commands from the current environment state, and robot-frame actions remove the need for precise camera-to-end-effector calibration.The network is trained end to end using this servoing mechanism.Source: A. Overview · p. 3
• Each time step records overhead and wrist images, motor positions, the chosen action, and the resulting reward as training data.The policy uses the current state to select the gripper’s next movement and evaluates the action with the reward function.Source: A. Overview · p. 3
• The action space comprises end-effector displacement along three axes, z-axis rotation, and gripper closure, which terminates the episode.Source: A. Overview · p. 3
• The Grasp-Q-Network combines visionNet and motorNet and estimates Q-values for each action as a Q-function approximator for the visual-servoing policy.Repeatedly using the target network to generate data for refitting the GQN is described as fitted Q iteration.Source: A. Overview · p. 3

B. DQN Algorithm

• The DQN returns action values from a state and observation sequence, and its algorithm uses a target network and experience replay.Source: B. DQN Algorithm · p. 3
• The target network shares the GQN architecture, with its parameters copied from the GQN every N time steps and held fixed between copies.Source: B. DQN Algorithm · pp. 3–4
• An action that is not executed receives reward −1 before the algorithm returns to the current-state step.Source: B. DQN Algorithm · p. 4
• A successful pick receives +10, a partial pick +1, and an unsuccessful result −0.025.Source: B. DQN Algorithm · p. 4

Y DQN

• γ is the discount factor and lies in [0, 1).Source: Y DQN · p. 4
• Values of γ closer to 0 indicate action choices based only on the current reward.Source: Y DQN · p. 4
• Values of γ approaching 1 indicate action choices based on long-term reward.Source: Y DQN · p. 4

C. Training

• Two value functions are trained symmetrically, with experiences assigned arbitrarily to update one function at a time.One weight set learns the greedy policy, while the other determines its value for each update.Source: C. Training · p. 4
• The target-value description identifies r_t as the immediate reward and s_{t+1} as the resulting state.Source: C. Training · p. 4
• Q(s, a | φ) is the parameterized value function used in the target-value description.Source: C. Training · p. 4

Y DDQN

• Double Q-learning solves the overestimation issue faced with Q-learning and outperforms the existing Deep Q-learning algorithm.Source: Y DDQN · p. 4
• Double DQN uses a Bellman equation to minimize the loss function L(φ) while training a policy network to approximate the optimal policy.The action selected by the current network is used by the max operator in double Q-learning.Source: Y DDQN · p. 4

D. Reward Function

• A successful grasp earns 10 when the gripper holds an object above a specified height at episode end; contact earns 1, no action earns -1, and each pre-termination step incurs -0.025.Source: D. Reward Function · p. 4

V. EXPERIMENTS

• The experiments compare proposed deep Q-learning and double deep Q-learning against vanilla Q-learning in simulation and on the real-world 7-DOF Baxter, with single- and multiple-camera setups.Source: §V · p. 4

A. Setup

• The simulation uses ROS with Gazebo and Atari DQN; a Baxter with parallel-plate gripper acts on randomly spawned cubes, spheres, and cylinders, while the real-world setup uses overhead and wrist cameras.The real-world overhead camera position was adjusted to avoid robot collision; Fig. 3 illustrates simulated and real-world grasping.Source: A. Setup · p. 4

B. Training

• The supplied figures depict Baxter learning to grasp in simulation and the real world and show learning curves for Q-learning, DQN, and double-DQN.Source: Figure 3 · p. 5, Figure 5 · p. 5
• The training objects are shown as 3D models, and the training setup uses spheres, cylinders, and cubes placed at random orientations.Source: B. Training · p. 4, Figure 4 · p. 5
• Arm movement is limited to wrist and shoulder rotation and reach extension, with the gripper facing downward; gripper feedback determines grasp success and object contact earns partial reward.Source: B. Training · p. 5
• Overhead- and wrist-camera images, together with motor positions, enter the Grasp-Q-Network, which sends servo commands as the robot moves toward the object and closes its gripper.The episode ends after a successful pickup or when the maximum step count is reached.Source: B. Training · p. 5

C. Results

• DQN and double-DQN had comparatively higher completion and success rates than baseline Q-learning, and double-DQN required the fewest steps for successful completion across 7,000 episodes.The experiments also report that the method can be adapted to the real robot by adjusting the robot software configuration.Source: C. Results · p. 5
• 91.1% multi-view accuracy and 83.2% single-view accuracy were achieved on the cube, the best-performing object for both models.The multi-view model performed better than the single-view model for the sphere and cylinder as well; the sphere had the lowest success rate.Source: Figure 6 · p. 6, C. Results · p. 6

VI. CONCLUSIONS

• The conclusion reports successful operation in simulation and on the real robot, with continuous feedback used to correct inaccuracies and adjust the gripper to object movement.Source: §VI · p. 6
• The conclusion states that the off-policy method performs better than vanilla Q-learning and reduces the overestimation observed with deep Q-learning.Source: §VI · p. 6