PaperlayerView exact revision on arXiv

Source-linked AI summary

Iterative Grasp Pose Refinement: A Deep Reinforcement Learning Approach for 2D Vision

Amir Arsalan Nematollahi, Shayan Ahmadi, Mehdi Tale Masouleh, Ahmad Kalhor

arXiv:2608.17628v1cs.ROcs.AIcs.LGeess.IV

TL;DR

Geometric grasping methods can overlook viable grasp points and lack adaptability, motivating a reinforcement-learning framework that combines keypoint representations with DQN-based iterative grasp refinement. The framework achieved a 100% success rate across 38 objects and successfully transferred a refined grasp to a physical robot.

  • Problem

    Geometric grasping methods can overlook viable grasp points, while keypoint-based approaches remain constrained by manual 3D annotation and limited adaptability.

  • Method

    The framework combines keypoint-based object representations with a DQN that iteratively refines grasp configurations through discrete pose adjustments.

  • Results

    100% success was achieved across 38 objects, with grasp configurations succeeding in all three evaluation trials; a refined grasp also produced a stable physical lift.

  • Takeaways & Limitations

    The findings support reinforcement learning as a grasp-refinement strategy for objects initially labeled ungraspable by geometric methods, including sim-to-real validation.

Abstract

from arXiv · show

Developing robots capable of understanding and manipulating objects requires compact, interpretable, and generalizable representations. This work proposes a reinforcement learning-based framework for robotic grasp refinement, integrating keypoint-based object representations with a Deep Q-Network (DQN). Using 2D overhead images captured in a simulated environment, a geometric-based algorithm generates initial grasp candidates, which are iteratively refined by the proposed framework, transforming failed grasps into successful ones. Experiments conducted on 300 objects from the Dex-Net dataset using a UR5 manipulator demonstrate the framework's effectiveness, achieving a 100% success rate on objects previously deemed ungraspable by geometrical methods. The framework's sim-to-real transferability is further validated through physical experiments on a Delta parallel robot, where a refined grasp successfully manipulates an object that was previously ungraspable. The findings underscore the effectiveness of reinforcement learning in addressing challenges in robotic grasping, offering a scalable and adaptable solution for contact-rich manipulation tasks.

I. INTRODUCTION

The introduction frames robotic grasping as requiring more dexterity and adaptability, motivating interpretable keypoint representations combined with reinforcement learning. It presents a DQN-based refinement framework that iteratively adjusts failed geometric grasp candidates in 2D images.

  • Motivation: Robotic manipulation remains challenging because current systems lack the human-like dexterity and adaptability required in unstructured environments.Deep reinforcement learning is motivated as a data-driven approach that learns grasping strategies through trial-and-error interaction.
  • Object Representation: Keypoint representations provide compact, structured object geometry that can support precise manipulation and generalization across diverse objects.The introduction contrasts this interpretability with black-box end-to-end representations and notes scalability barriers from manual 3D annotation.
  • Related Work: Reinforcement-learning grasping remains challenged by sample efficiency and sparse rewards, motivating approaches such as demonstration-guided learning and graph-based architectures.The introduction identifies supervised learning and reinforcement learning as the primary data-driven grasping pipelines.
  • Proposed Framework: The proposed DQN refines failed grasp candidates generated by a geometric algorithm from 2D overhead images by iteratively adjusting (x, y, θ, w).The framework bridges traditional geometric methods and data-driven learning by focusing reinforcement learning on grasp refinement rather than initial grasp learning.
  • Proposed Framework: Focusing reinforcement learning on local optimization around promising candidates makes grasp refinement more tractable, efficient, and scalable than global exploration.The methodology is presented as a refinement module that can convert failed or suboptimal grasp attempts into successful, stable configurations.

II. DATASET GENERATION AND LABELING · A. Simulation Environment and Configuration

The section describes a simulation-based pipeline for dataset generation and object keypoint labeling. It uses CoppeliaSim with a UR5 manipulator, RG2 gripper, and overhead RGB-D sensing to model robotic grasping tasks.

  • II. DATASET GENERATION AND LABELING: The section covers the simulation environment, dataset processing, and methodology for assigning object keypoints.These components form the basis of the dataset generation and labeling process.
  • A. Simulation Environment and Configuration: CoppeliaSim was selected as the simulation platform for its robust and feature-rich capabilities.The environment supports the simulation setup used for robotic grasping.
  • A. Simulation Environment and Configuration: The simulation environment provides precise physics-based modeling for robotic grasping tasks.Physics-based modeling is identified as a key capability of the selected platform.
  • A. Simulation Environment and Configuration: A UR5 manipulator equipped with an RG2 gripper was used in the simulation.The robot and gripper constitute the simulated manipulation hardware.
  • A. Simulation Environment and Configuration: An RGB-D vision sensor captured images of the virtual workspace from an overhead view.The sensor was integrated into the simulation environment for workspace observation.
  • A. Simulation Environment and Configuration: The overhead camera perspective provided a comprehensive view of the simulated workspace.This viewpoint supported visual observation during the simulated grasping setup.

B. Object Dataset · C. Initial Labeling

The study uses 300 diverse Dex-Net object meshes in CoppeliaSim, where a geometric algorithm generates and tests initial grasp candidates. Of 52 objects without successful initial configurations, 38 are selected for grasp refinement after excluding 14 beyond the RG2 gripper’s opening limit.

  • B. Object Dataset: The dataset is a random subset of Dex-Net containing 300 diverse object meshes imported into CoppeliaSim for simulation.The objects formed the evaluation set for the grasping study.
  • B. Object Dataset: A geometric algorithm proposed by Sabzejou et al. evaluates possible grasp configurations to initially label the objects.The initial labels identify grasp candidates based on object geometry.
  • B. Object Dataset: The initial grasp labels are tested in the simulator to determine whether their configurations produce successful grasps.Testing follows the initial labeling stage.
  • B. Object Dataset: 52 objects lacked any successful grasp configuration during initial evaluation.These objects were identified as failures of the initial geometric evaluation.
  • B. Object Dataset: 14 of the 52 objects were excluded because their geometries exceeded the RG2 gripper’s maximum opening width.The remaining objects proceeded to the refinement study.
  • B. Object Dataset: 38 remaining objects each contributed one randomly selected failed grasp candidate to the proposed refinement algorithm.The selection tested whether refinement could convert failed candidates into successful grasps.
  • C. Initial Labeling: Overhead RGB-D images captured in CoppeliaSim serve as input to the geometric algorithm for identifying initial grasp configurations.The camera records the scene before geometric labeling.
  • C. Initial Labeling: Initial grasp candidates are evaluated for stability and lift success, and their scores provide a baseline for subsequent refinement.The scores quantify the performance of each tested configuration.

III. GRASP REFINEMENT FRAMEWORK · A. Overview

The framework iteratively refines an initial grasp pose in simulation to convert failed grasps into successful, stable configurations. Reinforcement learning adjusts grasp parameters and uses stability-based scoring to guide this refinement.

  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: The method starts from an initial grasp pose and systematically modifies its dimensions in the simulation environment.The initial pose serves as input to the refinement model.
  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: The refinement process targets both failed grasps and already successful grasps whose stability can be improved.Its primary objective is converting failed grasps into successful ones while enhancing grasp quality.
  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: Object position and orientation are recorded before and after the grasp attempt to define initial and final stages.The initial stage provides a baseline for comparison.
  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: Reinforcement learning iteratively adjusts the grasp pose (x, y, θ, w) to maximize stability and minimize slippage.The initial grasp candidate is generated through geometric labeling before refinement.
  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: The scoring mechanism evaluates each state and serves as the reward guiding the agent toward optimal grasp configurations.The workflow uses reinforcement learning to select pose adjustments.
  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: A successful outcome lifts the object to an appropriate height with negligible slippage or instability.This outcome is one of the possible grasp results evaluated by the framework.
  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: The simulator returns a success flag and a stability score after each grasp attempt using a modified grasp pose.These outputs provide feedback for subsequent refinement steps.
  • III. GRASP REFINEMENT FRAMEWORK · A. Overview: Iterative refinement seeks grasp configurations that achieve the highest simulator-returned scores.The model uses repeated pose modification to identify stronger configurations.

B. Scoring and Reward Structure

The scoring system evaluates grasp stability through rotational slip between initial and final object orientations. Scores are normalized by outcome, rewarding tilt-free successful grasps and partially successful failures that produce less severe instability.

  • Scoring: Rotational slip (|∆θ|) is computed from differences between the object’s initial and final roll, pitch, and yaw orientations.The maximum possible angular deviation is π radians.
  • Reward Structure: +1 denotes a perfectly stable, tilt-free grasp, while successful grasps receive scores in [0, 1] and failures in [−1, 0].Failure is identified when the gripper fully closes without an object between its fingers, typically because the object drops or cannot be lifted.
  • Reward Structure: Among successful grasps, smaller rotational differences indicate greater stability and precision.Minimizing slippage is essential for stable and secure grasps.
  • Reward Structure: Within failed grasps, scores approaching 0 reward contacts that partially move or lift the object, even if it later falls.Greater object motion or rotation indicates a failed grasp closer to success.

C. Actions and States Definition

The framework represents each grasp configuration as a four-parameter state and refines it through discrete, incremental actions, including a no-op option.

  • State Definition: The grasp state is a 4D vector (x, y, θ, w) encoding a rectangle’s center coordinates, in-plane rotation, and width for a two-finger gripper.These four continuous parameters define the grasp configuration.
  • Action Definition: Actions apply discrete, small adjustments to the state variables through (∆x, ∆y, ∆θ, ∆w).The predefined incremental changes enable exploration and refinement of grasp configurations.
  • Action Definition: A no-op action leaves the grasp state unchanged when the network determines that no modification is needed.It is included alongside the discrete adjustment actions.

D. DQN Implementation and Training

The refinement framework uses a value-based DQN to approximate Q-values for local grasp refinement in a low-dimensional state space. It represents grasp configurations with four variables, selects among nine discrete adjustments, and trains from stability-based rewards using replay, function approximation, and ε-greedy exploration.

  • DQN selection and formulation: A value-based DQN approximates the Q-value function for computationally efficient local refinement in the problem’s low-dimensional state space.The network approximates Q(s, a | ϕ), where ϕ denotes the network parameters.
  • State, action, and reward: The state is a 4D grasp configuration vector st = (x, y, θ, w), while each action is one of 9 discrete adjustments.Actions are selected from a predefined action space comprising (∆x, ∆y, ∆θ, ∆w).
  • State, action, and reward: The immediate reward is a stability score ranging from [−1, +1], calculated from the object’s rotational slip after a grasp attempt.Training samples mini-batches of experiences (s, a, r, s′) from an experience replay buffer and minimizes the Mean Squared Error (MSE) loss.
  • Network and training configuration: The Q-network has three fully connected hidden layers with 256, 128, and 64 neurons, using ReLU activations and Layer Normalization.It uses Adam with a learning rate of 0.005 and a discount factor γ set to 0.9.
  • Network and training configuration: An ϵ-greedy policy starts with ϵ = 1.0, decays by 0.999 per step, and reaches a minimum of 0.01.This policy provides exploration during training while progressively reducing the exploration rate.

IV. RESULTS AND DISCUSSION

Across 300 Dex-Net objects, the refinement framework converted all 38 geometrically ungraspable cases into successful grasps, with repeated trials confirming reliability. DQN training progressively improved rewards and grasp configurations, and a refined grasp transferred successfully from simulation to a Delta robot experiment.

  • Dataset and evaluation: 300 Dex-Net objects were evaluated, excluding 14 physically ungraspable objects and identifying 38 remaining objects without successful geometric grasp configurations.The 14 exclusions were due to physical limitations such as gripper maximum opening width; the 38 objects lacked grasps enabling lifting to the required height.
  • Grasp-refinement performance: 100% success was achieved on all 38 objects previously lacking successful geometric grasp configurations.Each grasp was executed three times and counted as successful only when all trials succeeded.
  • DQN training progression: Average rewards increased steadily during training, while grasp configurations progressed from initial unsuccessful candidates toward improved and refined poses.Initial candidates produced minimal rewards; later configurations showed improvements, and highest-reward episodes yielded refined grasps meeting success criteria.
  • DQN training progression: The evolving rewards and grasp poses demonstrated reinforcement learning’s capability to improve grasp quality and address objects initially deemed ungraspable by geometric algorithms.The reported progression supported the framework’s robustness and adaptability in grasping tasks.
  • Sim-to-real validation: A refined grasp that succeeded in simulation was transferred to a Delta parallel manipulator and produced a successful, stable lift in the physical experiment.The corresponding initial failed grasp also failed on the real manipulator before refinement.

V. CONCLUSION

The framework combines keypoint representations with a DQN to iteratively refine geometrically generated grasp poses, converting failed candidates into stable configurations. It achieves 100% success on 38 hardware-feasible objects initially labeled ungraspable and validates sim-to-real transfer with a Delta robot.

  • Framework: The proposed framework combines keypoint-based object representations with a Deep Q-Network to iteratively optimize grasp configurations beyond geometric labeling.A geometric-based labeling algorithm provides the baseline, while reinforcement learning refines initially failed grasps into stable and precise poses.
  • Simulation results: 100% success was achieved for all 38 objects within hardware limits among 52 objects initially labeled ungraspable.The framework systematically explored and refined grasp configurations for objects challenging because of their geometry or orientation.
  • Simulation results: Progressively increasing rewards validated the DQN’s ability to transform low-reward failures into optimized grasp configurations.This result supports the framework’s systematic refinement process for difficult grasp scenarios.
  • Sim-to-real validation: A refined pose enabled a stable lift of a pair of pliers with a Delta parallel robot after the initial geometric candidate failed in simulation and reality.The refined pose had first been identified as successful in simulation, providing sim-to-real validation of the framework.
  • Implications: The findings contribute toward more scalable and versatile robotic systems for robust manipulation in complex, real-world environments.The conclusion frames the framework as a potential solution for challenging manipulation scenarios.
Loading 2608.17628v1…