Source-linked AI summary
Relative Contact Velocity-Controlled Hand-Object Mechanism for Dexterous Tool Manipulation
Sunyu Wang, Jean Oh, Nancy S. Pollard
TL;DR
General multi-finger robotic hands need to perform the full sequence of picking up, loading, and wielding a tool. The paper models the hand and tool as a hand-object mechanism with generalized contact frames and sub-assembly constraints, then evaluates the framework in simulation. All five tested hands completed the sequence with dexterous behavior from identical, simple reference trajectories.
Problem
The paper asks how general multi-finger robotic hands can perform the complete pick-load-wield tool manipulation process.
Source: §I · p. 1Method
Results
All five robotic hands completed the tool manipulation process in simulation, showing dexterous behavior from identical, simple reference trajectories despite differing kinematics and geometries.
Source: §I · p. 2Takeaways & Limitations
The results demonstrate the framework’s effectiveness and adaptability across different hands in simulation.
Source: §V · pp. 7–8In autonomous settings, reaching high-quality pregrasp poses is non-trivial, and the framework relies on suitable starting pregrasp poses.
Source: E. Limitations and Future Work · p. 7
Abstract
from arXiv · showhide
This work investigates how to enable general multi-finger robotic hands to perform the complete tool manipulation process, which entails picking up a tool, loading it into a suitable pose, and then wielding it. Inspired by human tool manipulation and mechanical design principles, we model the hand and the tool as a unified hand-object mechanism (HOM) composed of sub-assemblies. Specifically, we define a HOM as consisting of the hand, the object, and the generalized contact frames, allowing the HOM's motions to be expressed with the same set of Cartesian-space relative contact velocities, irrespective of the hand's kinematics and geometry. Then, we define a HOM's sub-assemblies as relative contact velocity and contact force constraints between fingers. Building on these definitions, we developed a lightweight and physically interpretable motion planning and contact estimation framework using least squares and a complementary filter. We evaluated our framework in simulation by teleoperating five different robotic hands. The results show that our framework enabled all five hands to execute the complete tool manipulation process, achieving dexterous behaviors even from identical, simple reference trajectories. Furthermore, the results showcase our framework's adaptability to different hands, tools, and tasks, enabled by its kinematic and geometric foundation. Please visit website for videos: \href{https://sunyuw.github.io/hom-sim/}{https://sunyuw.github.io/hom-sim/}
I. INTRODUCTION
The framework addresses the lack of a general, concrete hand-object mechanism (HOM) definition for controlling multi-finger hands through the complete pick-load-wield process. It draws on human manipulation concepts and enables five simulated hands to complete that process.
I. INTRODUCTION
- Generalized contact frames express dexterous in-hand manipulation through common Cartesian-space relative contact velocities, while sub-assemblies couple finger velocities and forces.The framework uses least squares for motion planning and a complementary filter for contact estimation.Source: §I · p. 2
- All five robotic hands completed pick-load-wield tool manipulation in simulation, showing dexterous behaviors from identical, simple reference trajectories despite differing kinematics and geometries.The framework is presented as adaptable to different hands, tools, and tasks.Source: §I · p. 2
A. Hand-Object Mechanism (HOM)
- Prior HOM work left general HOM definitions and control of general multi-finger hands unanswered; this work defines HOMs and their constituent sub-assemblies.Workspace studies used HOM mainly as qualitative inspiration, while hand-design studies tied HOMs to particular morphologies.Source: A. Hand-Object Mechanism (HOM) · p. 2
B. Human Hand Manipulation Studies
- Unlike human-hand studies that analyze and classify human manipulation, this work takes a synthesis approach to enable general robotic hands to perform pick-load-wield manipulation.Its objective is not a comprehensive analysis of human hand behaviors.Source: B. Human Hand Manipulation Studies · p. 2
C. Robotic Manipulation Motion Planning and Control
The work takes a bottom-up, mechanism-based approach to manipulation planning, drawing on human tool manipulation to identify reusable finger sub-assemblies. These sub-assemblies include pinches and pushers whose functions depend on contact configuration and force.
C. Robotic Manipulation Motion Planning and Control
- The framework aligns with optimal control through rigid-body roll-slide contact kinematics and a least-squares-based policy, while its bottom-up HOM approach is described as lighter and easier to set up across robots and tasks.The paper contrasts this with top-down end-to-end approaches that can rely heavily on data and computational resources.Source: C. Robotic Manipulation Motion… · p. 2
A. Mechanical Intuition and Identification of Sub-Assemblies from Human Tool Manipulation
- The human hand is treated as a mechanism approximator that forms mechanisms with tools, and sub-assemblies are the manipulation “tokens” sought as atomic units.This intuition draws on evidence that hand behavior operates through postural synergies and grasp and movement types rather than individual degrees of freedom.Source: A. Mechanical Intuition and… · pp. 2–3
- A pinch can immobilize a tool, permit in-hand rotation or translation, or actuate axial rotation depending on contact locations and forces.The pinch is a two-finger antipodal grasp; offset contacts can act like a double rack-and-pinion mechanism.Source: A. Mechanical Intuition and… · p. 3
- A pusher uses one finger’s motion and contact-force direction to actuate or support a tool, often pairing with a pinch to move the tool about or along its joint.A pusher can also press the tool against the environment to form a one-finger pinch.Source: A. Mechanical Intuition and… · p. 3
B. Definitions
A hand-object mechanism comprises the hand, object, and generalized contact frames, which connect them as a parallel mechanism. Relative contact velocities describe each finger’s motion against the object, and sub-assemblies add constraints coupling fingers.
B. Definitions
- A HOM consists of the hand, the object, and generalized contact frames, assuming nominally rigid bodies and known hand and object kinematics and surface geometries.The frames kinematically connect hand and object as one parallel mechanism, whether or not contact is present.Source: B. Definitions · p. 3
- Generalized contact frames use time-updated Cartesian frames on each finger body and the object, with origins and normal axes determined by whether the bodies are in contact.Without contact, the origins are closest-surface witness points; in contact, they are tied to the instantaneous contact point.Source: B. Definitions · p. 3, Figure 2 · p. 3
- Each finger’s relative contact velocities describe rolling, spinning, sliding, and separating from or approaching the object along the contact-frame axes.The tangent-plane components represent roll and slide; the normal-axis components represent spin and separation.Source: B. Definitions · p. 3
- A sub-assembly adds constraints coupling particular fingers; for a pinch, the fingers’ separation velocities match and their contact forces have equal magnitude and opposite directions.The opposed forces apply zero net wrench to the object and achieve force closure when the line between contact points lies within their friction cones.Source: B. Definitions · pp. 3–4
C. Motion Planning with Relative Contact Velocities
The planner converts relative contact-velocity or object-twist references into finger motion through least squares, while accommodating joint limits and stabilizing commands. A reduced-order pinch-frame speed lets a pusher rotate the object about the pinch axis.
C. Motion Planning with Relative Contact Velocities
- The motion pipeline substitutes the finger-body Jacobian into the contact-velocity relation and treats the object twist as an input when solving for finger commands.The joint velocities are integrated into reference joint positions, then tracked using inverse dynamics.Source: C. Motion Planning with Relative… · p. 4
- The planner solves for reference joint velocities from relative contact velocities while respecting joint position and velocity limits.Damping stabilizes commands near singularities, a healthy-pose term steers redundant fingers away from difficult-to-exit regions, and separation commands are clamped near the object to prevent penetration.Source: C. Motion Planning with Relative… · p. 4
- Finger motion can be commanded with either relative contact-velocity references or a reference object twist set according to the finger’s sub-assembly.The object twist lets fingers manipulate the object as the HOM’s effector with simple commands, regardless of finger kinematics and geometry.Source: C. Motion Planning with Relative… · p. 4
- A single reduced-order speed commands object rotation about the pinch axis, treating the pinch as a revolute joint and the pusher as its actuator.The pinch frame is fixed to the object, with its origin at the midpoint between contact points and its x-axis parallel to their connecting line.Source: C. Motion Planning with Relative… · p. 4
D. Contact Estimation for Stable Impedance Control
A complementary filter turns the generalized contact frame’s noisy origin into a continuous impedance-controller action point. Its gain balances smooth motion at low reference geodesic speeds with proximity to the instantaneous contact point at high speeds.
D. Contact Estimation for Stable Impedance Control
- The filter treats the generalized contact frame’s origin as a noisy measurement and estimates a continuous action point for stable impedance control.Its process model uses rigid-body roll-slide contact kinematics and geodesic tracing.Source: D. Contact Estimation for Stable… · p. 4
- The surface frames move on the object and finger body, with their normal axes aligned to the respective outward-pointing surface normals.The reference geodesic velocities are obtained from roll-slide contact kinematics with normal-axis angular and linear elements omitted.Source: D. Contact Estimation for Stable… · p. 4
- The filter gain supports finer integration at low reference geodesic speeds and keeps the estimated contact frame closer to the instantaneous contact point at high speeds.The threshold α is the geodesic speed above which the gain exceeds 95% of its maximum value.Source: D. Contact Estimation for Stable… · p. 4
- Using the estimated finger-side frame’s origin and normal as the action point and direction lets the impedance controller apply contact forces stably.The frame’s origin and normal are unsuitable for direct use because the instantaneous contact point is sensitive to minute contact movement.Source: D. Contact Estimation for Stable… · p. 4
E. Software Implementation and Experimental Setup
The framework was implemented in Drake with hydroelastic contact modeling and an approximate geodesic tracer. A slider-based teleoperation interface was used to evaluate five different hands in four experiments.
E. Software Implementation and Experimental Setup
- Drake’s hydroelastic contact model was selected because the pinch sub-assembly requires frictional torque that point-contact models struggle to handle.Contact surfaces were enabled by tetrahedralizing the relevant triangular meshes; contact points and generalized contact frames were then estimated from pressure centers and geometric queries.Source: E. Software Implementation and… · p. 5
- The contact-estimation pipeline approximates geodesics by tracing an osculating-circle arc, which is close to true geodesic tracing when paths are short, as they generally are here.The osculating circle uses vertex-interpolated principal curvature values and directions.Source: E. Software Implementation and… · p. 5
- The teleoperation interface provides per-finger sliders for sub-assembly selection, force magnitude, and relative contact-velocity references, plus object-twist, object-speed, and palm-twist controls.The simulation ran at 1 kHz and the framework at 500 Hz; visualization ran at under 20% of real-time speed.Source: E. Software Implementation and… · p. 5
- The evaluation teleoperated five hands, from commercial models to custom three- and five-finger designs, across four experiments using the same weights for every hand.The hands included Orca hand V2, Wuji hand, Sharpa Wave, custom A, and custom B.Source: E. Software Implementation and… · p. 5
A. Moving Around a Cylinder Without Contact
In the cylinder trajectory experiment, all hands achieved similar Cartesian-space trajectories despite different kinematics, with tracking quality varying by axis. Coupled angular and linear motion also let a single commanded velocity trace around the cylinder and switch sides.
A. Moving Around a Cylinder Without Contact
- The experiment sequentially commanded step trajectories for each scalar relative contact velocity while finger 2 moved around the cylinder without contact.The steps were 40 deg/s for angular velocities and 20 mm/s for linear velocities.Source: A. Moving Around a Cylinder Without… · p. 5
- All hands achieved similar Cartesian-space trajectories with decent tracking; y-axis angular and x-axis linear velocities tracked best, while x-axis angular velocity tracked worst.Fingers with fewer than six DoFs cannot satisfy all reference velocities simultaneously, and adduction or abduction for x-axis rotation induces other penalized velocities.Source: A. Moving Around a Cylinder Without… · p. 5
- A single non-zero z-axis linear-velocity reference enabled a finger to trace the cylinder at constant distance and switch sides relative to it.Translation along the generalized contact frame’s z-axis induces rotation about its y-axis.Source: A. Moving Around a Cylinder Without… · p. 5
B. Picking Up and Manipulating a Cylinder as Effector
The experiment tested whether different hands could pinch-lift and manipulate a cylinder using simple reference commands. All hands tracked cylinder motion reasonably, with rolling contacts supporting stable, dexterous manipulation.
B. Picking Up and Manipulating a Cylinder as Effector
- The hands pinched and lifted the cylinder, then rolled it along and about each body-frame axis while minimizing sliding and spinning.The experiment used simple step references for object body twist while setting relative contact velocities to zero.Source: B. Picking Up and Manipulating a… · p. 5
- All hands manipulated the cylinder as the hand-object mechanism’s effector, with reasonable tracking of object body linear velocity and mostly smaller-than-reference x- and z-axis angular velocities.The middle block of Fig. 4 reports object body velocity tracking.Source: B. Picking Up and Manipulating a… · p. 5, Figure 4 · p. 6
- Tracking the cylinder’s y-axis angular velocity was worse and produced x- and z-axis translation crosstalk because the pinch contacts could not rotate it about that axis.The fingers compensated for the requested rotation by translating the cylinder.Source: B. Picking Up and Manipulating a… · p. 6
- Noticeable relative contact angular velocities but small linear velocities along the cylinder’s y- and z-axes indicate rolling with little sliding during manipulation.This pattern is consistent with maintaining a stable grasp while moving the cylinder.Source: B. Picking Up and Manipulating a… · p. 6
- A simple z-axis angular-velocity step produced screwdriver-like twisting motions across hands with different kinematics and geometries.The cylinder’s body z-axis is its longitudinal axis.Source: B. Picking Up and Manipulating a… · p. 6
C. Rotating a Cylinder About the Pinch Axis and Forming a Tripod Grasp
The experiments used a pusher finger to rotate a pinched cylinder about the pinch axis, form a tripod grasp, and demonstrate screwdriver manipulation. The reported results show reasonable pinch-velocity tracking and a plug-and-play setup for a new hand.
C. Rotating a Cylinder About the Pinch Axis and Forming a Tripod Grasp
- Finger 3 entered a pusher sub-assembly to rotate the cylinder about the revolute joint formed by fingers 1 and 2.This rotation is described as a motion commonly used by humans to load a tool.Source: C. Rotating a Cylinder About the Pinch… · pp. 6–7
- A 40 deg/s step reference for scalar s rotated the cylinder away from the palm; reversing the reference let gravity return it upright while finger 3 maintained contact.The finger then switched sides, repeated the cycle by pulling the cylinder toward the palm, and coordinated with the pinching fingers to form a tripod grasp.Source: C. Rotating a Cylinder About the Pinch… · p. 7
- Figure 4 (bottom) presents quantitative results; its object pinch velocities refer to the velocities in V b.Source: C. Rotating a Cylinder About the Pinch… · p. 7
- All hands reasonably tracked the reference object pinch velocities, with small pusher-finger contact linear velocities during cylinder rotation.This indicates that the pusher actuated the revolute joint formed by the pinch while rolling with little sliding.Source: C. Rotating a Cylinder About the Pinch… · p. 7
D. Picking Up, Loading, and Wielding a Screwdriver
- Each hand picked up, loaded, and wielded a precision screwdriver by aligning it against the table and repeating a retract-turn cycle.Opposite contacts limited the first turn’s range, prompting fingers 1 and 3 to retract and reposition before turning again.Source: D. Picking Up, Loading, and Wielding a… · p. 7
- The screwdriver demonstration showed dexterous manipulation across different hands, and configuring custom hand B through the demonstration took less than five hours.The framework requires the hand’s and object’s kinematics and geometries for setup.Source: D. Picking Up, Loading, and Wielding a… · p. 7
E. Limitations and Future Work
The framework’s demonstrations were in simulation, and high-quality manipulation requires suitable pregrasp poses. The conclusion reports success across five hands and identifies real-world contact sensing as future work.
E. Limitations and Future Work
- Suitable pregrasp poses are required, but reaching them autonomously is non-trivial because the planner is velocity-based under non-holonomic constraints.The experiments used heuristics to teleoperate the hands to pregrasp poses; grasp synthesis pipelines and human-hand grasp datasets are proposed as additions.Source: E. Limitations and Future Work · p. 7
- The framework’s effectiveness and adaptability were demonstrated in simulation; real-world implementation remains future work, with contact sensing expected to be the challenge.The authors plan to explore high-rate center-of-pressure estimation and sensor fusion with motion capture.Source: E. Limitations and Future Work · p. 7
V. CONCLUSIONS
- The framework treats the hand and tool as one parallel mechanism and controls fingers through sub-assemblies using generalized contact frames, least squares, and a complementary filter.The filter fuses contact point measurements into a process model based on rigid-body roll-slide contact kinematics and geodesic tracing.Source: §V · p. 7
- In simulation, all five hands manipulated a cylinder and completed screwdriver pick-load-wield demonstrations with dexterous motions.The conclusion says these results demonstrate effectiveness and adaptability across hands and motivate future real-world implementation.Source: §V · pp. 7–8