Source-linked AI summary
DEX-PHD: A Dexterous Portable Handheld Device for Multimodal Data Collection
Jooyoung Hong, Noboru Myers, Obin Kwon, Yixian Yang, Joohyung Kim
TL;DR
Dexterous manipulation data collection must capture robot-hand motion, contact, and proprioception, while existing approaches trade direct robot-state data for portability. DEX-PHD combines a handheld dexterous robot hand, continuous per-finger control, and synchronized multimodal logging. Across contact-rich tasks, physical open-loop replay preserved the main temporal structure of finger motions and task-relevant contact patterns.
Problem
Existing collection methods trade accurate robot-state acquisition for portable operation, and many handheld interfaces provide limited multi-finger control and synchronized hand-state or contact data.
Source: §I · p. 1Method
DEX-PHD integrates a dexterous robot hand into an untethered handheld device for continuous per-finger joint-level control and synchronized logging of hand data and wrist-camera video.
Source: §I · p. 1, §III · p. 2Results
Across peg-in-hole insertion, multi-object grasping, and drilling, open-loop physical replay preserved the main temporal structure of finger motions and task-relevant contact-pattern agreement.
Source: §V · p. 7Takeaways & Limitations
DEX-PHD provides a practical platform for untethered multimodal dexterous-manipulation data collection across diverse indoor and outdoor environments.
Source: §V · p. 7Open-loop replay leaves contact timing and fingertip force profiles variable with object pose, alignment, and whole-body motion errors; force signals are baseline-relative, and the interface targets six actuated degrees of freedom.
Source: §V · p. 7
Abstract
from arXiv · showhide
Existing manipulation data collection methods typically offer either accurate robot-state acquisition through kinematic-chain-based teleoperation or convenient, portable operation through vision-based approaches. We present DEX-PHD, a portable handheld device that combines these advantages for dexterous manipulation by integrating a dexterous robot hand, continuous per-finger control, and multimodal logging. The system records proprioceptive information, fingertip contact data, and wrist-camera video, while the operator directly controls the robot hand through joint-level mapping. Its untethered design enables data collection across diverse environments. We demonstrate DEX-PHD on contact-rich tasks requiring coordinated multi-finger manipulation and deploy the collected demonstrations through open-loop replay on a physical humanoid robot. The results show that open-loop replay preserves the main temporal patterns of finger motion and task-relevant contact. Therefore, DEX-PHD provides a practical platform for portable, multimodal dexterous manipulation data collection.
I. INTRODUCTION
Dexterous data collection must capture motion, contact, and proprioception, yet existing approaches trade direct robot-state data for portability. DEX-PHD combines direct per-finger robot-hand control with synchronized multimodal logging in an untethered handheld device.
I. INTRODUCTION
- Existing collection methods trade accurate joint-level robot data for portability, while handheld systems often lack coordinated multi-finger control and synchronized hand-state and contact data.Kinematic-chain teleoperation can require expensive, bulky, or fixed setups; vision-based methods rely on pose estimation and retargeting.Source: §I · p. 1
- DEX-PHD integrates a dexterous robot hand into a portable handheld device for continuous per-finger joint-level control and synchronized multimodal logging.It records wrist-view video, hand commands and states, and fingertip force signals during contact-rich manipulation.Source: §I · pp. 1–2
- DEX-PHD’s battery-powered, untethered design supports collection across diverse environments without an external computer or fixed robot platform.The device maps continuous trigger and knob inputs directly to robot-hand joints.Source: §I · p. 1
- The paper evaluates replayability on a physical humanoid robot using joint-trajectory tracking and task-relevant contact-pattern similarity.The replayability evaluation is presented as a contribution alongside the device and synchronized logging framework.Source: §I · p. 2
II. RELATED WORK
Prior dexterous data-collection interfaces often require specialized hardware, calibration, fixed platforms, or human-motion retargeting; many handheld alternatives provide limited continuous multi-finger control. DEX-PHD combines direct robot-hand command generation with synchronized visual, proprioceptive, and contact data.
A. Dexterous Data Collection Interfaces
- Prior handheld collection systems use wrist-view, gripper-centric, tactile, force, or tool-based interfaces, but many provide limited continuous control of multiple robotic fingers.Handheld systems are presented as a portable alternative to teleoperation setups that may require specialized hardware, calibration, or a fixed platform.Source: A. Dexterous Data Collection Interfaces · p. 2
- DEX-PHD integrates a dexterous robot hand into a portable handheld device, enabling direct command generation without human-to-robot finger retargeting or a fixed platform.Many existing handheld systems focus on parallel grippers or tool-level interfaces and offer limited continuous control of multiple fingers.Source: A. Dexterous Data Collection Interfaces · p. 2
B. Multimodal Data Collection
- DEX-PHD synchronously records wrist-camera video, robot-hand commands and states, motor currents, and fingertip force signals during demonstrations.These signals capture interaction information that visual trajectories alone may not provide for contact-rich manipulation.Source: B. Multimodal Data Collection · p. 2
III. METHOD
DEX-PHD is a portable device for contact-rich dexterous demonstrations, combining continuous per-finger control with synchronized multimodal collection. Its handheld hardware places a robotic hand alongside a custom handle and records hand data and wrist-camera video onboard.
III. METHOD
- DEX-PHD supports continuous per-finger control, synchronized multimodal streams, and wireless collection beyond the lab without an external PC or fixed robot platform.The high-DoF hand provides tactile and proprioceptive data, while the wrist camera records the scene with time-aligned IMU measurements.Source: §III · p. 2
A. Hardware System
- The device is built around a 6-DOF robotic hand with a custom handle mounted alongside its dorsal side for portable, hand-centered operation.Independent flexion triggers control four fingers and thumb flexion; a dedicated knob controls thumb rotation.Source: A. Hardware System · p. 2
- The handle sits close to the hand to reduce collision footprint, and its bottom plane aligns with the palm for table-level reach.The aligned surfaces preserve clearance for planar manipulation so the hand can approach table-level objects without the handle contacting the surface first.Source: A. Hardware System · pp. 2–3, Figure 2 · p. 3
- Onboard electronics log proprioceptive states, user commands, and tactile signals from 30 FSRs at 100 Hz, while a wrist camera records at 60 fps.The system weighs 1.1 kg and has a practical runtime of 1.5 hours.Source: A. Hardware System · p. 3
B. Joint-Level Mapping
DEX-PHD calibrates continuous trigger and knob inputs, normalizes them, and linearly maps them to robot-hand motion ranges. Independent finger mapping supports coordinated manipulation, while logging captures commands and measured responses for replay and analysis.
B. Joint-Level Mapping
- Each input channel is calibrated from neutral and fully actuated configurations, normalized between those bounds, and linearly mapped to the corresponding robot-hand motion range.This produces joint-level commands from the operator’s continuous trigger and knob inputs.Source: B. Joint-Level Mapping · p. 3
- Four finger triggers independently command corresponding digit flexion, while a thumb trigger controls closing and a knob controls thumb rotation.The one-to-one mapping supports multi-finger reconfiguration, partial grasping, tool use, and sequential object release.Source: B. Joint-Level Mapping · p. 3
- The system logs command inputs, motor positions and currents, and fingertip forces alongside generated joint commands, enabling replay and analysis of commanded motion and measured response.Source: B. Joint-Level Mapping · p. 3
- The hand control and logging loop runs at 100 Hz, and observed finger responses occur in the same control cycle as input events within the logger’s 10 ms resolution.The estimated input-to-response delay is below or close to that measurement resolution.Source: B. Joint-Level Mapping · p. 3
C. Trajectory Generation
DEX-PHD estimates wrist trajectories from wrist-camera RGB and IMU data using visual SLAM. A scanned fiducial establishes a local base frame, which can be replaced by an offset transform when needed to keep robot trajectories reachable.
C. Trajectory Generation
- Visual SLAM estimates device pose over time from wrist-camera RGB and IMU data after a brief scan establishes a local base frame.The scan captures a base fiducial and scene keypoints for localization.Source: C. Trajectory Generation · p. 3
- The trajectory is transformed from the base-marker frame into the robot’s world frame using the marker-to-world transform.The passage identifies the fiducial marker, world, and end-effector frames as {B}, {W}, and {E}.Source: C. Trajectory Generation · p. 3
- When the marker’s world pose is uncertain or the trajectory is unreachable, the marker-to-world transform may be replaced with an offset transform.Source: C. Trajectory Generation · pp. 3–4
D. Data Synchronization
DEX-PHD supports collection across varied environments and physical humanoid replay, while demonstrations cover coordinated, contact-rich tasks. The paper also describes timestamp alignment for single-device recordings and bimanual data.
D. Data Synchronization
- LED blinks triggered at episode boundaries align hand and video streams by clipping video around the blink and pairing each frame with the nearest hand-data timestamp.Hand data are logged at 100 Hz and camera video at approximately 60 fps, giving nominal synchronization resolution on the order of tens of milliseconds.Source: D. Data Synchronization · p. 4
- For bimanual tasks, a rolling QR code maps camera-frame timestamps to unified system time for offline alignment of asynchronous streams.The QR sequence is displayed on a screen and captured by both cameras.Source: D. Data Synchronization · p. 4
IV. DEMONSTRATIONS
- The demonstrations evaluate untethered collection, coordinated multi-finger manipulation, and deployment across platforms, including on a physical humanoid.Open-loop replay was used on a 7-DoF-per-arm Unitree G1 and a 7-DoF dual-arm table setup in simulation, as well as a physical humanoid.Source: §IV · p. 4
A. Data Collection
- Demonstrations collected in diverse indoor and outdoor locations were subsequently replayed on a physical humanoid robot.The collection workflow used a battery-powered onboard logger, with setup and calibration each taking less than one minute.Source: A. Data Collection · pp. 4–5, Figure 3 · p. 4
- Peg insertion, multi-object grasping, and drilling probe fingertip contact capture and coordinated finger control under task-specific demands.Peg insertion has less than 0.5 mm clearance; drilling requires a stable power grasp with coordinated thumb rotation and trigger actuation.Source: A. Data Collection · p. 5
B. Inverse Kinematics
DEX-PHD uses embodiment-specific inverse kinematics to reproduce end-effector motions across robots with different configurations. Its closed-loop solver respects joint and motion constraints during replay.
B. Inverse Kinematics
- Because robot embodiments differ in degrees of freedom and physical dimensions, each requires its own inverse-kinematics solution for matching end-effector motions.Source: B. Inverse Kinematics · p. 5
- A closed-loop Pinocchio solver computes joint trajectories while enforcing joint limits and velocity and acceleration bounds.Joint commands are updated at each timestep and rate-limited to satisfy mechanical velocity constraints.Source: B. Inverse Kinematics · p. 5
- Embodiment-specific initial-pose offsets account for structural workspace differences and keep the motion within each robot’s reachable workspace.Source: B. Inverse Kinematics · p. 5
C. Cross-Embodiment Deployment
DEX-PHD wrist trajectories are deployed across robots with different kinematic configurations. In simulation, the same peg-in-hole demonstration produced consistent task behavior on dual-arm and humanoid platforms using embodiment-specific IK and offsets.
C. Cross-Embodiment Deployment
- Estimated wrist trajectories are deployed across platforms with different kinematic configurations using the embodiment-specific IK procedure.Source: C. Cross-Embodiment Deployment · p. 5
- Figure 4 covers wrist-trajectory generation and cross-embodiment deployment of DEX-PHD data.Source: Figure 4 · p. 5
- The same peg-in-hole sequence produced consistent task behavior on a dual-arm setup and humanoid robot despite their different kinematic structures.The replay used embodiment-specific inverse-kinematics solutions and offsets.Source: C. Cross-Embodiment Deployment · p. 5
D. Experimental Setup
The humanoid setup uses dedicated hand controllers and regulated power, and replay tests select one feasible, lowest-error demonstration per task. The results section compares replayed and recorded finger motion and contact using task-dependent measures.
D. Experimental Setup
- Two DC-DC converters reduce the humanoid’s 54 V supply to 7.4 V for the hands, and only its arms are actively controlled during deployment.The hand microcontrollers communicate with the embedded computer over USB using micro-ROS; the humanoid uses ROS 2.Source: D. Experimental Setup · pp. 5–6
- Ten episodes were collected per task, but only the feasible episode with the lowest end-effector tracking error was replayed once on the humanoid.Feasibility and tracking error were assessed under the embodiment-specific inverse-kinematics solution.Source: D. Experimental Setup · p. 6
E. Results
- Table II summarizes quantitative replay results, while Fig. 6 compares recorded and replayed joint trajectories and fingertip forces.Joint position MAE measures absolute tracking deviation; joint-position and contact correlations measure temporal similarity, with values nearer 1 indicating better agreement.Source: E. Results · p. 6, Table II · p. 6
- Contact signals are baseline-relative rather than absolute force measurements, and finger-level contact aggregates six FSR readings to reduce sensitivity to contact-location shifts.The initial non-contact interval supplies each trial’s per-channel baseline; contact correlation assesses task-relevant pattern similarity.Source: E. Results · p. 6
1) Peg-in-hole:
Peg-in-hole replay closely tracked finger motion but had low contact-pattern agreement, while multi-object grasping preserved grasp-and-release order and reproduced primary contacts.
1) Peg-in-hole:
- 2.66° mean joint MAE was the lowest among evaluated tasks in peg-in-hole, but its task-relevant contact correlation was 0.197.Joint trajectories generally followed the demonstration; motion agreement did not guarantee similar contact during open-loop insertion.Source: Figure 6 · p. 6, 1) Peg-in-hole · pp. 6–7
2) Multi-object grasping:
- Multi-object grasping achieved 5.48° mean joint MAE and 0.691 joint-position correlation, with 0.691 contact correlation for the index finger and thumb.Replay preserved the timing and order of grasp and release events; secondary contact profiles varied with cup placement and fingertip contact location.Source: 2) Multi-object grasping · p. 7
3) Drilling:
Drilling replay retained the main timing of grasp and trigger actuation despite pose and contact variation, while replay analysis covered only in-air drilling. Across tasks, demonstrations preserved finger-motion structure, with open-loop and interface limitations remaining.
3) Drilling:
- 7.54° was the largest joint MAE among tasks, while drilling had the highest joint-position correlation at 0.803 and contact correlation of 0.761.Replay preserved the main timing of grasp and trigger actuation despite variations in object positioning, tool weight, manipulator sagging, and torso orientation.Source: 3) Drilling · p. 7
- Humanoid replay analysis focused on in-air drilling because drill weight and whole-body posture constraints made the full fastening sequence difficult to reproduce reliably.The complete pick-up, screw-fastening, and tool-replacement task was demonstrated with DEX-PHD.Source: 3) Drilling · p. 7
V. CONCLUSION
- DEX-PHD combines continuous per-finger joint-level commands with synchronized wrist-camera, hand-state, and fingertip-force logging in a battery-powered, onboard system.The untethered design supports collection across indoor and outdoor environments without a fixed robot platform or external PC.Source: §V · p. 7
- Across peg-in-hole, multi-object grasping, and drilling, replayable demonstrations preserved the main temporal structure of finger motion, with task-dependent tracking and contact agreement.The paper suggests synchronized hand-state and contact signals can provide useful evaluation signals for contact-rich manipulation.Source: §V · p. 7
- Open-loop replay allows contact timing and fingertip-force profiles to vary with object pose, alignment, and whole-body motion; force signals are baseline-relative, not calibrated absolute measurements.The current interface is designed for six actuated hand degrees of freedom; support for hands with more independently actuated joints is future work.Source: §V · p. 7