Source-linked AI summary
Force-Modulated Dexterity: Unified Compliance for Contact-Rich Manipulation
Ying Yuan, David Held, Jitendra Malik, Haozhi Qi
TL;DR
Teleoperators can specify motion but struggle to reproduce subtle contact-force adjustments, and dexterous manipulation requires coordinating motion and force. FOMO-Dex selectively modifies nominal commands using force feedback in task- and contact-relevant directions. Across four tasks it reduces excessive forces, while its demonstrations support improved policy outcomes and successful demonstration collection.
Problem
Teleoperators struggle to reproduce subtle force adjustments without feeling robot contacts, although dexterous manipulation requires coordinating motion and contact forces.
Source: §1 · p. 2Method
FOMO-Dex uses task and contact geometry to select directions where force feedback modifies nominal commands, leaving the remaining components unchanged.
Source: §1 · p. 2, §5 · p. 14Results
Across four tasks, FOMO-Dex reduces excessive contact forces versus stiff control; screwdriving rotation increases from 4.2 to 34.7 turns per episode, with fewer tool slips.
Source: §1 · p. 2Takeaways & Limitations
The results suggest explicit force modulation provides a practical interface between motion commands and contact responses, and force-modulated demonstrations benefit behavior-cloning policies.
Source: §5 · p. 14The described setting is teleoperation of a robotic arm and multi-fingered hand through human motion tracking.
Source: §3.1 · p. 5
Abstract
from arXiv · showhide
Teleoperation is widely used to collect high-quality demonstrations for robot learning. Dexterous manipulation, however, requires coordinated control of motion and contact forces. Operators can specify motion, but struggle to reproduce the subtle force adjustments of human manipulation without feeling the robot's contacts. We introduce Force-Modulated Dexterity (FOMO-Dex), a unified compliance formulation for contact-rich manipulation. FOMO-Dex expresses nominal actions in coordinates defined by task geometry and contact structure. It applies force-driven motion corrections along selected directions while leaving the remaining command components unchanged. Across card picking, paper cutting, screwdriving, and nut-and-bolt assembly, FOMO-Dex reduces excessive contact forces compared with stiff control. A user study shows that operators collect more successful demonstration data per hour after brief practice. For policy learning, imitating the corrected actions improves card-picking success over imitating raw operator commands. Policies trained on FOMO-Dex demonstrations also achieve more rotation in nut-and-bolt manipulation and screwdriving than those trained on demonstrations collected with the stiff controller. Videos are shown on https://yingyuan0414.github.io/fomo-dex/ .
1. Introduction
FOMO-Dex addresses the difficulty of reproducing contact-force adjustments through teleoperation by selectively modifying nominal commands according to task and contact geometry. Across four tasks and in policy learning, it reduces excessive forces and improves manipulation or demonstration outcomes.
1. Introduction
- Teleoperators struggle to reproduce subtle force adjustments without feeling robot contacts, and small contact errors can produce large, inconsistent forces.The introduction frames the contribution as a unified interface, task-specific compliance realizations across four tasks, and paired nominal/corrected actions for policy learning.Source: §1 · pp. 2–3
- FOMO-Dex defines interaction directions from task geometry and contacts, converts force errors along them into motion corrections, and leaves excluded command components unchanged.Corrections can act through the arm or fingers, using wrist force-torque, tactile, or joint-torque feedback appropriate to the contact.Source: §1 · p. 2
- Across four tasks, FOMO-Dex reduces excessive contact forces versus stiff control; screwdriving rotation rises from 4.2 to 34.7 turns per episode, with fewer tool slips.Operators also collect more successful demonstration data per hour after brief practice.Source: §1 · p. 2
- In card picking, imitating force-modulated actions improves lift success over imitating raw commands with force modulation applied at deployment.Policies trained on FOMO-Dex demonstrations also achieve more rotation in nut-and-bolt manipulation and screwdriving than policies trained on stiff-control demonstrations.Source: §1 · p. 2
2. Related Work
Prior work addresses dexterous teleoperation, compliance, and learning compliant manipulation through different interfaces and control targets. FOMO-Dex applies robot-side force feedback to nominal arm and finger commands, with task-configured corrections and corrected commands for policy learning.
2.1. Dexterous Teleoperation
- Dexterous teleoperation interfaces map tracked human motion to robot hands, but motion tracking alone does not regulate contact forces; haptic systems and wearable exoskeletons offer other ways to convey or feel contact.Robot-side compliance instead adjusts contact response; FOMO-Dex focuses on fingertip loading and arm-mediated tool contact while leaving unselected command components unchanged.Source: §2.1 · p. 3
2.2. Compliance and Hybrid Position-Force Control
- Hybrid position-force, impedance, admittance, and task-frame methods provide established ways to relate motion, force, and interaction geometry.Unlike object-level impedance approaches that control desired object motion through the fingers, FOMO-Dex corrects nominal arm and finger commands without requiring an object-pose reference; it builds on established task-frame and admittance principles.Source: §2.2 · pp. 3–4
2.3. Learning Compliant Manipulation
- Learning-compliance methods estimate or predict compliance alongside motion, while other approaches learn interaction frames or force-aware residual policies.FOMO-Dex specifies force-response directions and feedback laws from task configuration rather than requiring a learned compliance or residual model.Source: §2.3 · p. 4
- Prior force-informed policy methods reconstruct targets from measured motion and force or learn mappings from predicted state and tactile trajectories; FOMO-Dex directly imitates corrected commands sent to the robot.Its system overview shows nominal arm and finger commands being adjusted using wrist force-torque and hand tactile and joint-torque feedback.Source: §2.3 · pp. 4–5, Figure 2 · p. 4
3.1. Problem Formulation
The problem formulation considers teleoperated arm and multi-fingered-hand commands alongside wrist and hand force feedback. It aims to regulate contact forces toward task-specific targets or limits while retaining commanded motion in directions that should remain stiff.
3.1. Problem Formulation
- The formulation considers a robotic arm and multi-fingered hand teleoperated through human motion tracking, with nominal commands and force feedback from the wrist and hand.The stated scope is this teleoperation setting.Source: §3.1 · p. 5
- Directly executing nominal commands during contact can produce excessive force, unstable tool motion, or grasp loss; the goal is force regulation without suppressing motion in directions that should remain stiff.Force may be regulated toward task-specific targets or limits.Source: §3.1 · p. 5
3.2. System Setup
The system pairs a 7-DoF arm with a 22-DoF dexterous hand and uses wrist and hand sensing for interaction feedback.
3.2. System Setup
- The setup uses a 7-DoF Tianji Marvin arm, a 22-DoF Sharpa Wave hand, a wrist force-torque sensor, and fingertip tactile and joint-torque measurements.A Sharpa Avatar glove tracks finger motion, while an HTC Vive Tracker attached to the glove tracks hand pose.Source: §3.2 · p. 5
3.3. Unified Force-Modulation Controller
FOMO-Dex adds force-derived corrections to nominal robot commands in task- and contact-aligned directions, leaving unselected command components unchanged. The same interaction-space approach can be applied through the arm or fingers and configured for distinct contact conditions.
3.3. Unified Force-Modulation Controller
- FOMO-Dex combines a nominal command with a force-derived correction, using task geometry and current contacts to select where modulation acts.The remaining command components retain their nominal values.Source: §3.3 · p. 5
3.3.1. Task-Frame Compliance Representation
- Task geometry and contact structure define unit interaction directions, and directions excluded from force modulation receive no compliance-induced correction.These directions identify motion components relevant to the interaction.Source: §3.3.1 · p. 5
3.3.2. Force Error and Admittance Modulation
- Selected directions use signed target-force errors or unilateral errors for limiting excessive loads, which a direction-wise admittance law converts into motion corrections.Admittance parameters determine the response; the scalar correction state is reset when contact or modulation status changes.Source: §3.3.2 · p. 6
3.3.3. Mapping to Arm and Hand Commands
- The controller maps interaction-space corrections through the arm’s Cartesian equilibrium pose or through fingertip displacements mapped to finger-joint commands.Both mappings apply the same corrections through the respective command interface.Source: §3.3.3 · p. 7
3.3.4. Task-Specific Compliance Configuration
- Card picking uses unilateral finger retreat, cutting uses lateral and surface-normal arm compliance, and rotational manipulation regulates fingertip forces while preserving commanded rotation.For cutting, the cutting direction is excluded from modulation; rotational manipulation uses fingertip contact normals and force targets derived from a shared grip level.Source: §3.3.4 · p. 8
3.4. Visuomotor Policy Training
The policy-training section describes behavior cloning from force-modulated demonstrations. The policy uses visual, proprioceptive, and force inputs and learns the corrected joint commands sent to the robot.
3.4. Visuomotor Policy Training
- Behavior cloning evaluates whether collected demonstrations support autonomous manipulation.The policy receives head-camera RGB images, robot proprioception, and force measurements.Source: §3.4 · p. 8
- A frozen pretrained DINOv2 encoder extracts image features, and a diffusion policy predicts joint-space actions.Proprioceptive inputs are randomly masked during training to encourage use of visual and force observations.Source: §3.4 · p. 8
- Training targets are the force-modulated joint commands sent to the robot, incorporating operator motion and force-dependent corrections rather than raw nominal commands.Source: §3.4 · p. 8
4. Experiments
The experiments assess FOMO-Dex in teleoperation and autonomous policy execution. The stated questions concern contact forces and task progress, successful-demonstration throughput across users, and comparison with stiff control.
4. Experiments
- The experiments evaluate how force modulation affects contact forces and task progress during teleoperation.Source: §4 · p. 8
- The evaluation also considers autonomous policy execution alongside teleoperation.Source: §4 · p. 8
- A user-study question asks whether FOMO-Dex increases successful-demonstration throughput across users relative to stiff control.Source: §4 · p. 8
3) How do demonstration collection and action targets affect autonomous policy performance?
The evaluation compares FOMO-Dex with stiff and naive-compliance controllers across four contact-rich tasks, then tests demonstration throughput and behavior-cloning outcomes. Results show force reductions, improved successful-data collection after practice, and better policy outcomes from corrected actions or FOMO-Dex demonstrations.
4.2. Controller Analysis on Teleoperation Data
- Across four tasks, FOMO-Dex reduces excessive forces while preserving task progress better than stiff control and naive compliance, which can interfere with task completion.In paper cutting, it reduces lateral and normal loads while maintaining comparable cut rate; in screwdriving, it improves rotation and engagement.Source: Table 2 · p. 9, §4.2 · pp. 10–11, Figure 4 · p. 10
4.1.1. Baselines
- The stiff controller directly executes operator commands, while naive compliance applies the same admittance parameters isotropically instead of selecting task- and contact-relevant directions.Controllers share the teleoperation interface, command rate, sensing, low-level control, and safety limits.Source: §4.1.1 · p. 9
4.1.2. Benchmark Problems
- The four benchmarks are card pickup, paper cutting, bolt–nut screwing, and screwdriving, with task progress and force-related measures evaluated for each.Measures include pickup success, cut rate, completed turns, slips, interaction forces, and force exceedance.Source: §4.1.2 · pp. 9–10
4.1.3. Trial protocol
- Trials reset objects and the robot to a common initial configuration, record commands and contact measurements at 30 Hz, and keep excessive-force thresholds fixed during collection.The user study additionally uses randomized task and controller order, with 10 evaluation trials before and after 10 minutes of practice.Source: §4.1.3 · p. 10, §4.3 · p. 12
4.2. Controller Analysis on Teleoperation Data
- Force modulation changes only contact-relevant hand motion: a surface-contact fingertip retreats, while active index–thumb contacts adjust without changing the remaining grasp configuration.Figure 6 contrasts the nominal teleoperator command with the force-modulated command.Source: Figure 6 · p. 11, §4.2 · pp. 11–12
4.3. User Study
- After practice, FOMO-Dex increases successful-data throughput across card picking, bolt–nut manipulation, and screwdriving; screwdriving episodes also last longer than with stiff control.Bolt–nut throughput is similar zero-shot, while FOMO-Dex improves throughput in card picking and screwdriving both zero-shot and after practice.Source: §4.3 · p. 13, Table 3 · p. 13
4.4. Behavior-Cloning Results
- For card picking, imitating force-modulated actions yields the highest lift success, while raw-command training produces no successful lifts even with force modulation at deployment.The comparison uses 50 demonstrations per task–controller setting and 18 evaluation trials.Source: Table 4 · p. 13, §4.4 · p. 14
- Policies trained on FOMO-Dex demonstrations achieve substantially more rotation than stiff-demonstration policies in bolt–nut manipulation and screwdriving.The cross-task comparison evaluates turns per episode over 10 trials.Source: Table 4 · p. 13, §4.4 · p. 14
5. Conclusions
FOMO-Dex selectively adjusts contact-relevant components of nominal actions using force feedback in task- and contact-relevant coordinates. Across four contact-rich tasks, it improves contact behavior while preserving task progress, supports successful demonstration collection and policy learning, and motivates broader future evaluation.
5. Conclusions
- FOMO-Dex treats teleoperator or policy output as a nominal action and selectively adjusts contact-relevant components using force feedback in task- and contact-relevant coordinates.The framework is presented as a unified approach to force-modulated dexterous manipulation.Source: §5 · p. 14
- Across four contact-rich tasks, FOMO-Dex improves contact behavior while preserving task progress, increases successful demonstration throughput in a user study, and enables behavior-cloning policies to benefit from force-modulated demonstrations.These results suggest explicit force modulation provides a practical interface between motion commands and the contact responses required for dexterous manipulation.Source: §5 · p. 14
- Future work will infer interaction directions and force targets online, incorporate uncertainty-aware contact estimation, and evaluate the framework across broader tasks and objects.Source: §5 · p. 14