Real-Time Grasping Strategies Using Event Camera

1 Introduction

• Model-based methods generally require prior object knowledge, whereas model-free methods can handle known and unknown objects by learning geometric parameters from vision.The framework assumes no obstacles between the objects and gripper, and addresses localization, pose estimation, and grasp generation.Source: §1 · pp. 2–3
• Event cameras generate asynchronous events at microsecond-level sampling, while frame cameras can suffer motion blur, low sampling rates, and limited low-light performance.Faster sensing can reserve more time for gripper actuation in closed-loop grasping.Source: §1 · pp. 2–3
• Prior event-based grasp generation cited in the introduction classifies poses as good or bad, requiring iterative adjustment rather than directly producing a grasp pose.Its LED marker frequency is constrained to 1 kHz.Source: §1 · p. 4
• The proposed work develops event-based instance segmentation for multiple objects and depth-aware visual servoing, addressing challenges in extending prior single-object grasping to multiple 3D objects.It also develops grasp generation from segmented spatial-temporal events using a Barrett hand.Source: §1 · p. 4
• The paper develops model-based and model-free event-based approaches for grasping multiple objects in cluttered scenes.The work evaluates the approaches with a neuromorphic eye-in-hand manipulator and Barrett hand across objects of varied sizes and geometries.Source: §1 · pp. 4–5

2 Overview of the proposed approaches

• An event is generated when the logarithmic brightness change at a pixel exceeds threshold C; its stream has microsecond temporal resolution and includes spatial address, timestamp, and polarity.The study uses a DAVIS 346 camera with 140 dB dynamic range, 10 mW power consumption, and 346 × 260 resolution.Source: §2 · p. 5
• Both approaches obtain 3D spatial and centroid information from event data and depth estimation, then use object information and grasp planning for manipulation.The overview identifies gray blocks as shared processes.Source: §2 · pp. 5–6, Figure 1 · p. 6
• The model-based approach segments 3D point clouds and uses position-based visual servoing, while the model-free approach uses machine-learning instance segmentation and velocity-based visual servoing.The overview identifies green blocks with model-based processes and blue blocks with model-free processes.Source: §2 · pp. 5–6

3.1 Neuromorphic Multi-View Localization

• A single event camera follows a known trajectory and uses Event-Based Multi-View Stereo to estimate object pose by warping corner events into a Disparity Space Image and voting across voxels.The resulting depth map can be converted to a point cloud.Source: §3.1 · pp. 8–9

3 Model-based Grasping Approach

• The model-based workflow combines event-based localization, point-cloud processing, model registration, and robot manipulation with grasp planning.The camera is mounted on the manipulator in an eye-in-hand configuration.Source: §3 · pp. 6–7, §3.2 · p. 9

3.2 Point Cloud Processing

• Point-cloud processing downsamples the data, clusters objects, registers models, and supplies object poses for manipulation.The method describes these stages as the model-based multi-object grasping sequence.Source: §3.2 · p. 9

3.2.1 Point Cloud Downsampling

• A k-d tree nearest-neighbor search removes isolated points whose distances exceed a threshold.These isolated points are treated as likely outliers.Source: §3.2.1 · p. 9

3.2.2 Object Clustering

• Euclidean clustering groups the sparse corner-point clouds into individual object clusters using nearest neighbors within a threshold radius.The input contains at most eight corner points per object.Source: §3.2.2 · p. 9

3.2.3 Model Registration

• Registration uses an inexact eight-corner, 1 m cube model and SVD to estimate rotation, translation, and scale between clustered points and source-model points.The method addresses unseen corners and model-scale differences that original ICP does not handle.Source: §3.2.3 · p. 10

3.3 Robot Manipulation and Grasp Planning

• Position-based visual servoing guides the end-effector using the estimated 6DoF pose; inverse kinematics, planned joint trajectories, and PID control execute the motion.The final pose may be predefined or obtained from localization and point-cloud processing.Source: §3.3 · p. 11

4.1 Segmentation

• MEMS extends mean shift to asynchronous event streams by iteratively shifting spatiotemporal events toward their cluster’s distribution mode.The method uses both spatial and temporal information, unlike standard mean shift’s spatial-only treatment.Source: §4.1 · pp. 12, 14
• MEMS processes event data in 12.861 ms, compared with 754.977 ms for standard-vision mean shift.The paper presents soft speedup and event downsampling as further ways to accelerate segmentation.Source: §4.1 · p. 15

4.1.1 Strategy 1

• Strategy 1 performs better than standard mean shift on E-score for α < 0.75, especially at α = 0.35.Processing time per event and iteration declines through α = 0.35 and then rises; clustering metrics are slightly better at α = 0, with average differences around 1%. MEMS diverges at α = 1.Source: §4.1.1 · p. 16, Figure 5 · p. 16, Figure 6 · p. 17

4.1.2 Strategy 2

• At β = 2, MEMS strategy 2 reaches at least around 16% E-score improvement; processing time falls as β increases, reaching around 1 µs per event when β exceeds 4.At β = 1, recall, precision, and F1 outperform standard mean shift; the reported E-score improvement rises with β.Source: §4.1.2 · p. 17, Figure 8 · p. 18

4.2 Visual Servoing

• Event-based visual servoing uses MEMS centroids and depth to align the gripper through translational correction followed by angular correction.An eHarris detector extracts corners, and a mask and heatmap help isolate useful features and calculate a robust centroid.Source: §4.2 · pp. 18–20

4.3 Grasping Plan

• For unknown objects, PCA supplies a principal axis and centroid for grasp-pose generation, while a histogram of detected orientations selects a robust final rotation.The Barrett hand closes around the object until contact or joint limits, after checking that its fingers can achieve stable contact points.Source: §4.3 · pp. 20–21

4.4 Event-based Multiple Objects Grasping Framework

• The multi-object loop obtains depth once, freezes the initial depth map, then uses cluster IDs to align, grasp, and place objects in sequence.After each grasp, the ID increments and the gripper returns to its starting position for the next object.Source: §4.4 · p. 22

• The section validates multiple-object grasping experimentally and discusses the resulting performance.Source: §5 · p. 23

5.1 Experimental Setup and Protocol

• Experiments use a UR10 arm, Barrett hand, and eye-in-hand DAVIS346 event camera, with object size constrained by the gripper opening and manipulation range.The evaluation system estimates gripper and object poses from ArUco markers; its measured static-pose errors were 1° in angle and 0.1 cm in position.Source: §5.1 · pp. 23–24

5.1.1 Model-free grasping experiment protocol

• The model-free protocol explores depth once, segments objects with MEMS, then uses visual servoing and hand alignment before pick-and-place.Source: §5.1.1 · p. 25

5.1.2 Model-based grasping experiment protocol

• The model-based protocol scans the scene, localizes objects from corners, clusters point clouds, registers object models, and grasps using PBVS.Source: §5.1.2 · pp. 25–26

5.2 Evaluation Metrics

• The evaluation combines planned grasp-pose errors, post-grasp object deviations, success rate, and grasp quality score.Source: §5.2 · p. 26

5.2.1 Optimal Grasping Evaluation

• A grasp pose is successful only when position error egp is at most 2 cm and orientation error egr is at most 15°.The overall success rate is the proportion of trials meeting both limits.Source: §5.2.1 · p. 26

5.2.2 Object Deviation Evaluation

• Object deviation measures pose change from before to after grasping, and lower deviation yields a higher quality score within the predefined limits.Source: §5.2.2 · p. 27, Figure 16 · p. 27

5.3 Experimental Results and Analysis

• Both proposed neuromorphic vision-based grasping approaches successfully accomplished the grasping tasks, and all reported evaluation metrics were within their limitations, as shown in Tables 3 and 4.Source: §5.3 · p. 29

5.3.1 Robustness Testing

• In low light, both approaches remained within error limits and reached the grasping goal, though measured variability increased for aspects of each approach.The model-free approach had higher overall standard deviation; the model-based approach had higher standard deviation in object deviation.Source: §5.3.1 · p. 30, Table 3 · p. 30, Table 4 · p. 30
• Model-free grasping had smaller orientation error but larger position error than model-based grasping, while model-based had higher success rate and grasp quality in both lighting conditions.Source: §5.3.1 · pp. 30–31, Figure 19 · p. 31
• The model-free approach successfully pick-and-placed objects of different shapes, including unknown octahedrons, and was also validated on everyday objects.The model-based approach requires a prior object model, so it was not used for the unknown octahedrons.Source: §5.3.1 · pp. 31–32, Table 5 · p. 32
• The experiments demonstrate a successful pick-and-place task for multiple objects.Source: §5.3.1 · p. 33

6 Discussion

• The model-based approach shows slightly better grasping performance with less error, attributed to position-based visual servoing.Source: §6 · p. 33
• The model-based approach is limited to known objects requiring offline models, whereas the model-free approach can localize unknown objects without prior knowledge.Source: §6 · p. 33
• The model-free approach is less sensitive to imperfect perception and may also handle moving objects, unlike the model-based approach's stated known-object restriction.Source: §6 · p. 33

7 Conclusion

• The eye-in-hand framework combines 3D reconstruction, clustering, position-based visual servoing, and grasp planning for known objects with event-based segmentation, depth-informed visual servoing, and grasp planning for unknown objects.Source: §7 · p. 33
• Both approaches successfully perform multiple-object grasping in experiments with objects of different sizes and lighting conditions.Source: §7 · pp. 33, 35
• The model-based approach is slightly more accurate, while the model-free approach is more applicable to unknown objects in real scenarios.Source: §7 · p. 35
• The demonstrated multi-object grasping is limited to simple scenarios; handling occlusion is identified as future work for event-based object segmentation.Source: §7 · p. 35