Source-linked AI summary
Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision
Patrick Rosenberger, Akansel Cosgun, Rhys Newbury, Jun Kwan, Valerio Ortenzi, Peter Corke, Manfred Grafinger
TL;DR
Human-to-robot handovers require the robot to grasp objects from a person while maintaining safety and accommodating unfamiliar objects. This paper combines real-time perception and grasp selection with human-part filtering; it reports an 81.9% overall success rate across handovers involving 13 objects. The system is not yet ready for unsupervised practical use.
Problem
Human-to-robot handovers require the robot to grasp the object while ensuring safety, and useful systems should handle objects they have not seen before.
Source: §I · p. 1Method
The system combines object-independent real-time grasp selection with body-part and hand/finger segmentation that filters human pixels from candidate grasps.
Source: §I · p. 1, D. Grasp Selection · p. 5Results
81.9% of handovers succeeded across the 13 test objects.
Source: A. Handover Experiments · p. 5, B. Quantitative Results · p. 6Takeaways & Limitations
The approach selects grasps for previously unseen objects independently of orientation, presentation, and background, while segmenting human parts for safer handovers.
Source: §VII · pp. 6–7The system is not yet ready for practical use without supervision, and further development is required.
Source: §VII · p. 7
Abstract
from arXiv · showhide
We present an approach for safe and object-independent human-to-robot handovers using real time robotic vision and manipulation. We aim for general applicability with a generic object detector, a fast grasp selection algorithm and by using a single gripper-mounted RGB-D camera, hence not relying on external sensors. The robot is controlled via visual servoing towards the object of interest. Putting a high emphasis on safety, we use two perception modules: human body part segmentation and hand/finger segmentation. Pixels that are deemed to belong to the human are filtered out from candidate grasp poses, hence ensuring that the robot safely picks the object without colliding with the human partner. The grasp selection and perception modules run concurrently in real-time, which allows monitoring of the progress. In experiments with 13 objects, the robot was able to successfully take the object from the human in 81.9% of the trials.
I. INTRODUCTION
Human-to-robot handovers require partners to coordinate in real time despite differences in perception and uncertainty. This paper presents an object-independent approach that couples real-time perception and manipulation to grasp objects from people while avoiding human contact.
I. INTRODUCTION
- Safe and generalizable handovers require the robot to avoid human contact while adapting to objects it has not seen before.The partners’ workspaces must intersect for an exchange, so physical separation is not possible during handover.Source: §I · p. 1
- The system grasps previously unseen objects from a human partner while monitoring the person to avoid grabbing or pinching them.Its contributions include object-independent real-time grasp selection and real-time hand/finger detection.Source: §I · p. 1
- The approach was evaluated with 13 objects across 520 handover trials, with additional edge-case testing using more objects.The paper also reports a public release of the code.Source: §I · p. 1
II. RELATED WORK
Prior handover research has often focused on individual aspects of the exchange, while this work presents a full human-to-robot handover pipeline. It distinguishes its approach from methods that classify a limited set of grasp poses by selecting grasps from object shape and tracking the human partner.
II. RELATED WORK
- Previous work has focused primarily on individual aspects of human-to-robot handovers, whereas this work provides a full pipeline.The cited work spans coordination, handover prediction, touch-based grasping, imitation, social perception, and motion studies.Source: §II · p. 2
- Unlike Yang et al.’s [14] seven-pose classifier for grasping a held cube, this approach selects a picking location from object shape to transfer unknown objects.It also tracks the human body and hand to avoid unwanted contact during exchange.Source: §II · p. 2
III. DESIGN PRINCIPLES
The design principles prioritize safe, generalizable handovers that accommodate varied object presentations and environments. The system therefore detects human parts, handles unfamiliar objects and changing presentations, uses onboard sensing, and operates in real time.
III. DESIGN PRINCIPLES
- The robot should distinguish human parts from objects to minimize collision and pinching risks during handovers.Only low-impact incidental contact is allowed, and the robot should not pinch the human’s hands.Source: §III · p. 2
- Object independence requires handling previously unseen objects, supported here by a generic detector and learning-based grasp-pose selection.Systems limited to known objects are suited only to controlled environments.Source: §III · p. 2
- The robot should accept varied presentations, including objects on a palm, offered with both hands, or moving during handover.The approach is described as handling these presentation modes.Source: §III · p. 2
- To avoid reliance on environmental adaptations such as markers or external cameras, required sensing should be onboard.The system uses a single end-effector-mounted RGB-D camera and no artificial markers.Source: §III · p. 2
- Perception and grasping must run in real time so the robot can react to changes in the human hand and object positions.The paper also notes that real-time motion with deliberate delays can increase the perceived naturalness of handovers.Source: §III · pp. 2–3
IV. SYSTEM SETUP
The system uses a desk-mounted Franka-Emika Panda arm with a two-finger gripper and a gripper-mounted RGB-D camera.
IV. SYSTEM SETUP
- The setup pairs a 7-degree-of-freedom Panda arm with a two-finger gripper and an end-effector-mounted RGB-D camera.The camera captures 640×480 images at 30 fps; fully open, the gripper handles objects up to 7 cm wide.Source: §IV · p. 3
V. APPROACH
The approach organizes six modules into parallel processing within layers and sequential processing across layers, with event-driven inputs reducing image-to-arm latency. Object detection deliberately uses a low confidence threshold to cover objects beyond its 80 trained categories while filtering detections outside the workspace.
V. APPROACH
- The six-module pipeline processes images at 24–30 fps and takes 0.0625 seconds per image.Modules in the same layer operate in parallel, while different layers operate sequentially.Source: §V · p. 3
A. Object Detection
- YOLO v3 detects objects from RGB images, using a deliberately low confidence threshold to include objects outside its 80 COCO categories.The threshold is also high enough to prevent false detections of nonexistent objects.Source: A. Object Detection · pp. 3–4
- Fig. 3 shows YOLO detections with several objects on a table and with an object in the user’s hand.Source: Figure 3 · p. 4
- Detections outside the robot’s workspace and objects labeled “Person” are removed before selecting a target.The foremost object within reach is chosen; the robot waits to move until a graspable object is detected.Source: A. Object Detection · p. 4
B. Body Segmentation
The system pairs body-part segmentation with a separate hand segmentation module to identify human pixels, including fingers, for safer grasp selection. The hand model was validated on held-out EgoHands frames and also detects some human skin.
B. Body Segmentation
- The body segmentation module labels head, torso, arms, thighs, and lower legs, while a separate hand module targets hands and visible fingers.The separate module addresses the body network’s limitation in detecting fingers under occlusion.Source: B. Body Segmentation · p. 4, C. Hand Segmentation · p. 4
C. Hand Segmentation
- The hand network achieved 0.897 mIoU and 0.986 pixel accuracy on its validation set.EgoHands contains 48 first-person videos and 4,800 segmented frames; 4,400 were used for training and the rest for validation.Source: C. Hand Segmentation · p. 4
- The hand-only model also detects some human skin, which the authors retain because it does not affect system performance and contributes to safety.Source: C. Hand Segmentation · p. 4
D. Grasp Selection
Grasp selection adapts GG-CNN to hand-held objects by making the depth image resemble a tabletop scene and excluding human pixels and nearby grasp points. The highest-quality remaining grasp is transformed into the robot’s base frame.
D. Grasp Selection
- GG-CNN generates pixel-wise grasp positions, orientations, and quality estimates from depth images, and selects the highest-quality grasp for a two-finger gripper.The original network was trained on single objects lying on a table, unlike this hand-held-object setting.Source: D. Grasp Selection · p. 4
- An imaginary plane behind the target replaces the background, making the depth image resemble a single tabletop object and excluding pixels behind it.The plane is parallel to the camera’s optical axis.Source: D. Grasp Selection · p. 5
- Pixels identified as human by body and hand segmentation, plus grasp points within five pixels of a body part, are filtered out.The remaining grasp pose is transformed into the robot’s base coordinate frame before control.Source: D. Grasp Selection · p. 5
E. Robot Control
The robot driver selects a stable grasp pose, moves toward it through position-based visual servoing, and executes the grasp and drop. It monitors distance and human proximity, aborting or recovering when specified safety or controller errors occur.
E. Robot Control
- The handover point is the mean grasp location across five frames after frames deviating by more than 7 cm in any direction are discarded.The selected pose remains fixed until the handover is complete because of computational load and latency across ROS nodes.Source: E. Robot Control · p. 5
- Position-based visual servoing drives the robot to the selected grasp pose, where the gripper closes before the object is moved to a drop location.The gripper then opens over the drop location.Source: E. Robot Control · p. 5
- The driver aborts and restarts if object-distance deviation exceeds its margin or a human body part moves too close to the grasp point; controller errors or sensed collisions trigger a stop, gripper opening, and return home.The recovery behavior waits three seconds before returning the arm home.Source: E. Robot Control · p. 5
VI. PILOT STUDY
The pilot study evaluated human-to-robot handovers with 13 objects across 520 trials. Outcomes were classified by safe transfer, safety stop, grasping failure, or detection failure.
A. Handover Experiments
- The experiment used 13 test objects and 520 handovers to evaluate transfer from the human partner to the robot.The objects were YCB food items that fit the Panda gripper; four system developers each handed over every object 10 times.Source: A. Handover Experiments · p. 5
- Trials were labeled as safe transfer, observer-initiated safety stop, safe but unsuccessful grasp, or failure to detect the object within 30 seconds.Source: A. Handover Experiments · p. 5
- Each object was handed over 40 times, and Table I categorizes handover outcomes by object.Source: Table I · p. 6
B. Quantitative Results
The pilot achieved an 81.9% overall handover success rate, with grasping failure the most common failure mode. Further tests varied objects and interaction conditions; the authors identify grasping and object detection as limitations.
B. Quantitative Results
- 81.9% of handovers succeeded; grasping failed in 10.8%, safety stops occurred in 5.7%, and detection failed in 1.6% of trials.Grasping failure was the most common failure mode.Source: B. Quantitative Results · p. 6
- Success rates ranged from 75% for strawberry, biscuits, and jelly to 92.5% for spam and pear.Jelly was not detected consistently when oriented without distinguishing features visible to the camera.Source: B. Quantitative Results · p. 6
C. Qualitative Results and Future Work
- Internal tests with ten researchers varied user position, object properties, presentation, background, and hand speed using objects outside the training datasets.These tests were exploratory rather than conducted under a strict procedure.Source: C. Qualitative Results and Future Work · p. 6
- The authors identify grasping and detection as limitations, proposing real-time 3D object tracking and transfer-point updates, plus detection of hand and object intent and outline.They report that inconsistent detection can prompt users to move the object and wait longer for handover initiation.Source: C. Qualitative Results and Future Work · p. 6
VII. CONCLUSION
The conclusion presents an object-independent human-to-robot handover approach using real-time robotic vision. It handles previously unseen objects across variations in orientation, presentation, and background, while requiring further development before unsupervised practical use.
VII. CONCLUSION
- Object-independent grasp selection supports handovers of previously unseen objects regardless of orientation, presentation, or background.Source: §VII · pp. 6–7
- Deep-learning perception modules segment hands and human body parts to support safe handovers.Source: §VII · p. 7
- The authors report robustness in experiments and internal tests but state that further development is needed before practical use without supervision.They plan to incorporate the modifications outlined in Sec. VI-C.Source: §VII · p. 7