PaperlayerView exact revision on arXiv

Source-linked AI summary

Neuromorphic Eye-in-Hand Visual Servoing

Rajkumar Muthusamy, Abdulla Ayyad, Mohamad Halwani, Yahya Zweiri, Dongming Gan, Lakmal Seneviratne

arXiv:2004.07398v1cs.ROcs.CVcs.SYeess.SY

TL;DR

Frame-based visual processing can delay feedback for high-speed robotic tasks. The paper proposes eye-in-hand event-based visual servoing with layered event processing and switching control; experiments report a 100% success rate for randomly placed objects.

  • Problem

    Frame-based cameras have low sampling rates and process redundant image data, limiting continuous visual feedback in high-speed tasks.

    Source: §I · p. 1

  • Method

    The eye-in-hand method processes event streams through three active-event layers and switches control modes to explore, reach, align, and grasp.

    Source: §I · p. 2, A. Contributions · p. 2, A. Event Processing · p. 3

  • Results

    100% success rate was reported for grasping objects placed randomly in the workspace, without re-tuning across different geometrical features.

    Source: §IV · p. 8

  • Takeaways & Limitations

    The authors present the method as suitable for pick-and-place tasks in smart manufacturing applications.

    Source: §IV · p. 8

    The reported grasp errors were attributed mainly to camera-axis misalignment with the workspace plane and camera skew relative to the gripper center.

    Source: B. Experimental Results · p. 7

Abstract

from arXiv · show

Robotic vision plays a major role in factory automation to service robot applications. However, the traditional use of frame-based camera sets a limitation on continuous visual feedback due to their low sampling rate and redundant data in real-time image processing, especially in the case of high-speed tasks. Event cameras give human-like vision capabilities such as observing the dynamic changes asynchronously at a high temporal resolution ($1\mu s$) with low latency and wide dynamic range. In this paper, we present a visual servoing method using an event camera and a switching control strategy to explore, reach and grasp to achieve a manipulation task. We devise three surface layers of active events to directly process stream of events from relative motion. A purely event based approach is adopted to extract corner features, localize them robustly using heat maps and generate virtual features for tracking and alignment. Based on the visual feedback, the motion of the robot is controlled to make the temporal upcoming event features converge to the desired event in spatio-temporal space. The controller switches its strategy based on the sequence of operation to establish a stable grasp. The event based visual servoing (EVBS) method is validated experimentally using a commercial robot manipulator in an eye-in-hand configuration. Experiments prove the effectiveness of the EBVS method to track and grasp objects of different shapes without the need for re-tuning.

I. INTRODUCTION

Visual servoing uses visual feedback to control robot motion, but frame-based processing can delay robot action in high-speed tasks. This paper presents an eye-in-hand event-based approach that uses event streams to guide manipulation.

I. INTRODUCTION

  • Frame-based cameras process consecutive images, causing delays in visual processing and timely robot action.The introduction identifies this as a limitation for high-speed applications requiring fast, efficient, accurate, and reliable visual information.Source: §I · p. 1
  • Event cameras respond asynchronously to illumination changes, with low latency, high temporal resolution, and wide dynamic range.The paper exploits these sensor properties for visual servoing and robotic object manipulation.Source: §I · p. 1
  • The proposed eye-in-hand method controls robot motion from event streams generated by relative motion, tracking features toward a desired event-camera pose.The approach extracts, tracks, and matches event features such as points and lines.Source: §I · p. 2

A. Contributions

The paper introduces event-based visual servoing for eye-in-hand manipulation, with a switching strategy and three event-surface layers that process features and govern control modes.

A. Contributions

  • The contributions combine three active-event layers and a simple control law with explore, reach, and align switching for top-down vacuum-gripper grasping in a constrained 2D plane.The authors demonstrate the approach in an eye-in-hand configuration for manipulation applications.Source: A. Contributions · p. 2

II. EVENT-BASED VISUAL SERVOING METHOD

  • A double-loop scheme converts event features into a goal-versus-current error and camera velocity command, while a second loop stabilizes the manipulator joints.The switching strategy regulates feature streams across explore, reach, and align modes.Source: §II · p. 2

A. Event Processing

  • Three event-surface layers address the limits of isolated or all-history processing: SAE timestamps latest pixel events, SACE tracks corner events and a virtual object center, and SAVE supports mode switching.The object center is an extracted virtual event, while random and goal-state events are also represented as virtual events.Source: A. Event Processing · p. 3

B. Feature Detection

The method detects event corners, accumulates them in a time-varying heat map, and uses local peaks to estimate object features. The object centroid is then represented as a virtual event for tracking.

B. Feature Detection

  • The adapted e-Harris detector classifies event features and extracts corner events from local event information.The reported settings are HCth = 5, N = 20, and a 9 × 9-pixel patch.Source: B. Feature Detection · p. 3
  • A heat map accumulates consecutive corner events, with each event's influence scaled by α and spread according to σ.Its influence decays over time according to τ and the last corner-event timestamp.Source: B. Feature Detection · pp. 3–4
  • Object corners are detected as local heat-map peaks above 0.7 using a 10 × 10-pixel dilation window.Fig. 5 depicts a rendered event frame, a weighted-corners heat map, and feature events extracted in SACE and SAVE.Source: B. Feature Detection · p. 4, Figure 5 · p. 4
  • The centroid of the detected object is represented as a virtual event in SAVE for visual-servoing tracking.This centroid is computed from the local heat-map peaks.Source: B. Feature Detection · p. 4

C. Feature tracking

Feature tracking compares detected feature events with desired events and uses their error to control camera motion. Corner tracking uses a fast moving average, with periodic heat-map checks to detect tracking errors.

C. Feature tracking

  • The control objective is to reduce the pixel-coordinate error between detected high-level feature events and desired feature events.The feature Jacobian relates feature-event velocity to camera velocity, which serves as the robot controller input for exponential error decrease.Source: C. Feature tracking · p. 4
  • A moving average updates the tracked object corners as new e-Harris corner events arrive, refining the SAVE centroid.The approach is faster than heat-map corner detection and is suited to high-speed applications.Source: C. Feature tracking · p. 4
  • Moving-average tracking can err when a corner is lost, so heat-map detection checks tracking every 0.3 seconds and resumes after discrepancies across multiple timesteps.Source: C. Feature tracking · p. 4

D. Gripper Alignment to Grasp

After the robot reaches the object center, it adjusts gripper orientation for a stable grasp. The target angle is based on the object centroid and a selected corner, and the gripper rotates until that angle is admissible.

D. Gripper Alignment to Grasp

E. Switching Strategy

The switching strategy sequences exploration, reaching, and gripper alignment using virtual events. It can return to exploration when feature detection breaks down during reaching.

E. Switching Strategy

  • A random virtual event prompts workspace exploration, after which the robot tracks the detected object-center event once its contiguous-pixel count exceeds a threshold such as 3.The strategy then aligns the gripper for a stable grasp.Source: E. Switching Strategy · p. 5
  • If object-feature detection breaks down during reaching, the strategy can switch modes to recover and reach the desired feature.Source: E. Switching Strategy · p. 5
  • The algorithm initializes SAE, SACE, and SAVE, as well as a desired feature event in SAVE, before operating in an eye-in-hand configuration.Its stated output is camera velocity Vc = (υ, ω).Source: E. Switching Strategy · p. 5

3 Initialize switching strategy

The switching strategy processes event streams through corner detection, heat-map weighting, and centroid estimation, then changes behavior as object-feature contiguity develops. It proceeds from exploration and centroid tracking to gripper alignment and grasping.

3 Initialize switching strategy

  • The algorithm detects corners in the SAE, extracts weighted corners from SACE heat maps, and computes the object centroid.It then monitors and operates in SAVE.Source: §3 · p. 5
  • When the contiguity count is below Cth, the robot initializes a random desired event in SAVE and moves toward that feature.Source: §3 · p. 5
  • Once contiguity exceeds Cth, the algorithm tracks object-centroid events, aligns gripper orientation, and executes a pre-grasp move and grasp.Source: §3 · p. 5

III. EXPERIMENTAL VALIDATION OF EBVS

This section reports experiments conducted to validate the proposed EBVS approach.

III. EXPERIMENTAL VALIDATION OF EBVS

  • Experiments were conducted to validate the proposed EBVS approach.Source: §III · p. 5

A. Experimental Setup and Protocol

The experiments test a top-down grasping setup with an eye-in-hand event camera and multiple object geometries. The protocol uses planar exploration and reaching followed by axial approach, grasping, and relocation.

A. Experimental Setup and Protocol

  • The EBVS method was tested in a top-down grasping paradigm.Source: A. Experimental Setup and Protocol · p. 5
  • The setup uses a UR10 arm, custom vacuum gripper, and DAVIS240C event sensor in an eye-in-hand configuration.The sensor has 240 × 180 spatial resolution, 12-microsecond minimum latency, 12 MEvent/second bandwidth, and 120 dB dynamic range.Source: A. Experimental Setup and Protocol · p. 7
  • During exploration and reaching, the end-effector moves in a plane perpendicular to the camera optic axis, then translates along that axis to contact the object before grasping and relocation.The workspace was limited to a 1.2 x 1.0 m virtual rectangle because of UR10 reach and camera field-of-view limits.Source: A. Experimental Setup and Protocol · p. 7
  • The geometry evaluation used a triangular prism, cuboid, and pentagonal prism.Source: A. Experimental Setup and Protocol · p. 7

B. Experimental Results

Across the experiments, EBVS tracked and grasped the tested objects, with an average grasp error of 16.1 mm. The results also identify more frequent tracking loss on pentagonal shapes when camera motion is parallel to an edge.

B. Experimental Results

  • In the cuboid trial, exploration toward a random virtual event led to feature detection, centroid reaching, gripper alignment, and grasping and manipulation.Heat-map and top-view comparisons support the reported corner detection, tracking, and centroid inference.Source: B. Experimental Results · p. 7
  • The rectangular-shape experiment was repeated five times at different workspace placements, recording grasp error and tracking-loss switches.The grasp error is defined as the distance between the gripping-points center and the true object centroid.Source: B. Experimental Results · p. 7
  • Experiments across five trials covered three geometries: triangular prism, cuboid, and pentagonal prism.Source: B. Experimental Results · p. 7, Table II · p. 7
  • 16.1 mm was the average grasp error across experiments, while the method successfully tracked and grasped targets with both vacuum grippers adhering.The authors attribute errors mainly to camera-axis misalignment with the workspace plane and camera skew relative to the gripper center.Source: B. Experimental Results · p. 7
  • Pentagonal shapes produced more tracking losses and larger centroid deviations; edge-parallel camera motion can fail to trigger events needed for corner detection.The authors suggest filtering corners by reliability based on the camera velocity vector as a future development.Source: B. Experimental Results · pp. 7–8, Figure 10 · p. 8

IV. CONCLUSION AND FUTURE WORK

The study presents a purely event-based, multistage visual servoing method for pick-and-place, using active-event layers to drive gripper approach and alignment. Experiments report grasping objects of different geometries without retuning, with a 100% success rate on randomly placed objects.

IV. CONCLUSION AND FUTURE WORK

  • Three active-event layers process incoming events so the gripper can reach and align with a target for grasping and placement.The method detects and tracks high-level scene features for the pick-and-place task.Source: §IV · p. 8
  • The platform precisely grasped randomly placed objects with a 100% success rate and handled different geometries without re-tuning or adaptation.Experiments validated the method across objects with different geometrical features.Source: §IV · p. 8
  • Future work will account for alignment uncertainties and augment the procedure with optimal motion planning.Source: §IV · p. 8
Loading 2004.07398v1…