Source-linked AI summary
DexForce: Extracting Force-informed Actions from Kinesthetic Demonstrations for Dexterous Manipulation
Claire Chen, Zhongchun Yu, Hojung Choi, Mark Cutkosky, Jeannette Bohg
TL;DR
Contact-rich dexterous imitation requires actions that produce appropriate forces, but common demonstration methods are challenged by unintuitive human-to-robot retargeting and lack of haptic feedback. DexForce uses forces from kinesthetic demonstrations to compute force-informed actions; across six tasks, these actions yield an average success rate of 76%, compared with near-zero for actions that ignore contact forces.
Problem
Widely used retargeting-based demonstrations are difficult for contact-rich dexterous tasks because human-to-robot motion correspondence is unintuitive and direct haptic feedback is lacking.
Source: §I · p. 1Method
DexForce measures contact forces during kinesthetic demonstrations and uses them with observed robot motion to compute force-informed actions for policy learning.
Source: §I · p. 1, §III · p. 3Results
Takeaways & Limitations
DexForce provides a research tool for studying force sensing in imitation learning for contact-rich dexterous manipulation.
Source: §V · p. 7The authors acknowledge DexForce may not be best suited for large-scale data collection.
Source: §V · p. 7
Abstract
from arXiv · showhide
Imitation learning requires high-quality demonstrations consisting of sequences of state-action pairs. For contact-rich dexterous manipulation tasks that require dexterity, the actions in these state-action pairs must produce the right forces. Current widely-used methods for collecting dexterous manipulation demonstrations are difficult to use for demonstrating contact-rich tasks due to unintuitive human-to-robot motion retargeting and the lack of direct haptic feedback. Motivated by these concerns, we propose DexForce. DexForce leverages contact forces, measured during kinesthetic demonstrations, to compute force-informed actions for policy learning. We collect demonstrations for six tasks and show that policies trained on our force-informed actions achieve an average success rate of 76% across all tasks. In contrast, policies trained directly on actions that do not account for contact forces have near-zero success rates. We also conduct a study ablating the inclusion of force data in policy observations. We find that while using force data never hurts policy performance, it helps most for tasks that require advanced levels of precision and coordination, like opening an AirPods case and unscrewing a nut.
I. INTRODUCTION
DexForce addresses the difficulty of collecting demonstrations for contact-rich dexterous manipulation, where successful actions require appropriate forces. It uses forces measured during kinesthetic demonstrations to produce force-informed actions for policy learning.
I. INTRODUCTION
- AirPods-case opening and box flipping require precise forces at multiple contact points, while battery grasping and nut unscrewing also require coordinated contact making and breaking.These demands exceed the nuanced force application needed for pick-and-place tasks.Source: §I · p. 1
- Retargeted human demonstrations are difficult for dexterous tasks because human-to-robot motion correspondence is unintuitive and operators lack direct haptic feedback [17],.The human hand does not move like most robot hands, complicating precise motions and force application.Source: §I · p. 1
- DexForce computes force-informed actions from contact forces measured during kinesthetic demonstrations for learning dexterous manipulation policies.Kinesthetic teaching lets the operator use the robot hand's dexterity and receive direct haptic feedback.Source: §I · pp. 1–2
- 76% average success across six tasks, versus near-zero for policies trained on actions that do not account for contact forces.The study also finds that force observations improve performance across all six tasks, with the largest benefits for tasks requiring precision and coordination.Source: §I · p. 2
II. RELATED WORK
Prior dexterous-manipulation demonstration methods address different limitations of retargeting, force sensing, and task generality. DexForce uses kinesthetic teaching and force-informed actions to address contact-rich hand manipulation without assuming a fixed grasp.
II. RELATED WORK
- Retargeting-based collection lacks haptic feedback and faces the human-to-robot correspondence problem –[16].ResPilot and Tilde [17] mitigate correspondence challenges, but neither provides haptic feedback; Tilde also has high hardware overhead.Source: §II · p. 2
- Prior force-sensing hand methods assume non-slipping contact and mainly support fixed-grasp reorientation; DexForce handles contact switching without additional heuristics.One prior contact-switching example requires hand-designed, object-specific heuristics.Source: §II · p. 2
- Robot-arm force-reproduction methods are limited by absent vision inputs or task-specific control primitives –, whereas DexForce targets fine-grained robot-hand tasks.Related arm-policy work demonstrated door opening and vase wiping.Source: §II · p. 2
- Figure 2 illustrates that different applied forces can yield the same observed fingertip position, while force-informed target displacement scales with contact force.The figure visualizes −f, the force exerted by the object onto the finger.Source: Figure 2 · p. 3
III. METHOD
DexForce converts measured contact forces and observed fingertip positions into targets that can reproduce demonstrated forces under impedance control. Its two-stage procedure collects force-informed targets, replays them to obtain robot-only demonstrations, and uses those demonstrations for policy learning.
III. METHOD
- Cartesian impedance control tracks desired fingertip positions to produce motion and force; its control force combines position error scaled by stiffness and velocity damping.The controller applies the resulting force through finger joint torques using the Jacobian and gravity compensation.Source: §III · p. 3
- Force-informed targets combine observed fingertip positions and measured contact forces, using xf = xo + k_f f under a quasi-static assumption.The hand-tuned stiffness parameter k_f scales the force-related displacement; the model does not model soft contact or friction but was sufficient for many tasks.Source: §III · p. 3
- Stage 1 records fingertip-position and force trajectories from kinesthetic demonstrations and computes force-informed position targets.Six-axis force-torque sensors are mounted at each fingertip base.Source: §III · p. 3
- Stage 2 replays force-informed targets to create robot-only demonstrations matching test-time visual conditions, recording images and contact data for policy training.The learned policy receives image features and contact forces, outputs force-informed targets, and is supervised on those targets.Source: §III · pp. 3–4
IV. EXPERIMENTS
Experiments evaluate DexForce on six contact-rich tasks, comparing force-informed actions with observed fingertip positions and testing force observations and stiffness sensitivity. Force-informed policies outperform direct-position policies, while the observation and parameter studies examine where and how the method helps.
Experimental setup
- The six-task suite spans one- and two-finger contact-rich interactions; policies use matched 30-scene evaluations, RGB images, and thumb/index force-torque sensing.The Allegro Hand setup uses force-torque sensors at the thumb and index-finger bases; policies are CNN-based Diffusion Policies.Source: §IV · p. 4
Question 1: Force-informed actions
- 76% average success across six tasks for force-informed action policies, versus near-zero success for policies trained directly on observed fingertip positions.Direct-position policies often moved fingers to objects but failed to apply force; on unscrew nut, one achieved partial unscrewing in 23 of 30 trials but never grasped the nut to finish.Source: §IV · p. 5, Figure 7 · p. 6
Question 2: Force observations
- Force observations improve success most on open AirPods, unscrew nut, and slide cube; across all six tasks, RGB, 0/1 policies have lower success rates than RGB, F/T policies.RGB, 0/1 policies tend to perform equivalently to or worse than RGB-only policies.Source: Figure 8 · p. 6, §IV · pp. 6–7
- In OOD evaluations, policies perform worse than on in-distribution variants, yet complete tasks about 50% of the time, with RGB, F/T outperforming RGB-only.The OOD variants use nut rotations of 360°–900° and a flip box that is 100g heavier.Source: Figure 5 · p. 5, Figure 8 · p. 6, §IV · p. 7
Question 3: Stiffness sensitivity
- One shared kf range spans all six tasks, and kf = 0.0045 lies within each; unscrew nut, open AirPods, and slide cube have the narrowest ranges.Those three tasks also benefit most from force observations; the authors suggest that task force sensitivity and observation benefit are correlated.Source: §IV · pp. 6–7, Figure 9 · p. 7
V. CONCLUSION
DexForce collects demonstrations for contact-rich tasks that are difficult to demonstrate with widely used retargeting methods, and its force-informed actions support policy learning. The authors note that the approach may not suit large-scale data collection.
V. CONCLUSION
- 76% average success across six tasks was achieved by policies trained on force-informed actions, versus near-zero success for policies trained on actions that do not account for contact forces.The authors describe force-informed actions as valuable for policy learning.Source: §V · p. 7
- DexForce collects robot demonstrations for contact-rich tasks that are difficult to demonstrate with current widely used retargeting-based methods.Source: §V · p. 7
- DexForce may not be best suited for large-scale data collection.The authors present it as a tool for studying force sensing in imitation learning and say its insights may inform future data-collection methods.Source: §V · p. 7