PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Force Regulation for Whole-Hand Dexterous Grasping

Sang Min Kim, Alexander Alexiev, Tzu-Yuan Lin, Sangbae Kim, Young Min Kim, Yonghyeon Lee

arXiv:2609.30082v1cs.ROcs.SYeess.SY

TL;DR

Changing contacts make precomputed grasp forces unreliable, so the paper estimates whole-hand contacts geometrically and reallocates forces online without tactile sensing. In simulation, this outperforms non-adaptive and fingertip-only baselines; hardware trials show grasp maintenance and recovery after most losses.

  • Problem

    Contacts shift across the hand, changing the grasp and torque mappings and making a precomputed force distribution unreliable.

    Source: §I · p. 1

  • Method

    The method estimates contacts from tracked object geometry and hand proprioception, then recomputes forces over the evolving whole-hand contact set without tactile sensing.

    Source: §I · p. 2, B. Geometric Whole-Hand Contact… · p. 3, §V · p. 7

  • Results

    In simulation, whole-hand contacts outperform fingertip-only execution at every perturbation angle, reaching 88.6% versus 72.4% at 90°; on hardware, six of eight object losses led to autonomous regrasping.

    Source: 1) Dynamic Perturbation Resistance · p. 5, 3) Results · p. 7

  • Takeaways & Limitations

    Combined with reactive reaching, the controller runs autonomously from approach through grasp stabilization and can regrasp after most losses.

    Source: 3) Results · p. 7, §V · p. 7

    The controller depends on an object model and pose tracker, and its quasi-static force allocation does not explicitly compensate for gravity, inertia, or external loads.

    Source: §V · p. 7

Abstract

from arXiv · show

Robust dexterous grasping requires maintaining physical stability despite contacts interactively evolving across the entire hand. A precomputed force distribution can easily fail under object motion, modeling errors, or external disturbances. In this paper, we present a framework for real-time force regulation over dynamically changing whole-hand contacts. Our method geometrically estimates contacts across all hand links using a tracked object model and proprioception, without requiring tactile sensing at those contacts. It repeatedly recomputes the desired contact-force distribution subject to friction constraints, actuator limits, and an actuation-consistency constraint motivated by classical whole-limb force analysis. We integrate this force-regulation controller with reactive reaching, enabling the hand to acquire a grasp, maintain it under disturbances, and regrasp after losing the object. Simulation experiments without gravity demonstrate improved grasp retention over fixed-allocation and fingertip-only execution under controlled perturbations, while real-world experiments on a 27-DoF arm-hand system demonstrate grasp maintenance and recovery under human-applied disturbances as contacts evolve across the whole hand. Project page: https://sangminkim-99.github.io/reactive-grasp-whole-hand/

I. INTRODUCTION

Contacts shift across the whole hand, making precomputed force allocations unreliable and motivating continuous contact estimation and force reallocation. The paper uses geometry and proprioception to regulate those contacts without tactile sensing, addressing gaps in grasp execution and whole-hand contact estimation.

I. INTRODUCTION

  • Changing contact locations and normals alter the grasp matrix and contact Jacobian, so feasible force distributions require continuous contact estimation and reallocation.Contacts may differ from the plan from the start and keep appearing, disappearing, and shifting across the hand.Source: §I · p. 1
  • Geometry from a tracked object model and hand proprioception estimates contacts on any link without tactile sensing, enabling real-time force reallocation as contacts change.The allocation accounts for friction, actuator limits, and an actuation-consistency constraint motivated by whole-limb force analysis.Source: §I · p. 2, B. Contact State Estimation · p. 2

A. Dexterous Grasp Synthesis; B. Contact State Estimation

  • Prior grasp synthesis produces diverse poses, but execution is typically open-loop; contact-estimation alternatives provide only partial coverage or information.Tactile sensing may not cover the full articulated hand, joint-torque methods become ambiguous with multiple contacts, and learned estimators do not recover precise locations or normals.Source: A. Dexterous Grasp Synthesis · p. 2, B. Contact State Estimation · p. 2

C. Grasp Force Regulation

The paper regulates forces online as hand–object contacts evolve, combining current contact geometry with a force allocation over those contacts. Unlike approaches requiring tactile measurements at the contacts, it uses geometry and proprioception, while adopting a quasi-static formulation.

C. Grasp Force Regulation

  • The approach updates contacts across uninstrumented hand surfaces using geometry and proprioception, then includes them in online force allocation.This trades dedicated contact sensing for a known object model and tracked pose, and is demonstrated on a physical whole-hand system.Source: C. Grasp Force Regulation · p. 2

A. System Overview and Problem Formulation

  • At each control cycle, the controller estimates whole-hand contacts and recomputes forces, the net object wrench, and actuated-joint torques for the current contact configuration.No contact forces are measured; object motion redistributes forces over the updated contacts.Source: A. System Overview and Problem… · pp. 2–3
  • The quasi-static formulation seeks squeezing forces that resist unmodeled loads through friction rather than explicitly compensating for gravity, inertia, or external wrenches.The target object’s 6-DoF pose is continuously tracked, and its mesh comes from a CAD model in simulation or offline physical-object reconstruction.Source: A. System Overview and Problem… · pp. 2–3

B. Geometric Whole-Hand Contact Estimation

The controller searches the whole hand geometrically for contacts, then solves a constrained force-allocation problem over the current contact set. The optimization balances residual object wrench against friction and actuation constraints.

B. Geometric Whole-Hand Contact Estimation

  • The estimator transforms sampled hand-link points using the tracked object pose and joint configuration, then uses signed-distance queries to select contact candidates.It samples about 3000 candidate points over the hand, retaining up to K closest candidates per link below the distance threshold.Source: B. Geometric Whole-Hand Contact… · p. 3
  • Every link surface is checked each cycle, allowing contacts to appear anywhere on the hand as the object moves; the whole-hand search takes under 1 ms.Each contact is represented by its position, object-surface normal, and hand link.Source: B. Geometric Whole-Hand Contact… · p. 3

C. Real-Time Whole-Hand Force Allocation

  • Forces are recomputed over the current contact set, with the grasp matrix constructed from current contact positions and normals to map forces to the net object wrench.The allocation is posed as a convex quadratic program minimizing residual object wrench.Source: C. Real-Time Whole-Hand Force Allocation · p. 3
  • The force-allocation problem imposes unilateral contact and linearized friction constraints alongside minimum grasp-force, actuator-limit, and actuation-consistency constraints.Force and torque components use unit numerical weights in the implementation.Source: C. Real-Time Whole-Hand Force Allocation · p. 3, 1) Contact Constraints · p. 3

1) Contact Constraints:

The contact constraints include a conservative friction approximation and a minimum total normal force, while the method maps forces at actuated contacts to joint torques.

1) Contact Constraints:

  • The friction constraint uses a pyramid inscribed in the Coulomb cone, a conservative approximation, with µ denoting the static friction coefficient.The constraints also prevent the trivial zero-force solution.Source: 1) Contact Constraints · p. 3
  • The lower bound fn,min applies to total normal force; the paper sets fn,min = 10 N to require a squeezing distribution.The QP seeks a small residual wrench, achieving exact balance only when compatible with the constraints.Source: 1) Contact Constraints · p. 3
  • For contacts on actuated links, the corresponding force vectors are mapped to joint torques.Source: 2) Actuator Torque Limits · p. 3

2) Actuator Torque Limits:

Actuated-contact forces are mapped to joint torques, while unactuated contacts such as the palm contribute passive reactions. An actuation-consistency constraint limits desired forces that are invisible to finger-joint torques, and the controller repeatedly updates the optimization as contacts change.

2) Actuator Torque Limits:

  • Actuated finger-contact forces map to joint torques, while unactuated contacts such as the palm are treated as passive reactions.The corresponding actuated-contact Jacobians are stacked in Jact.Source: 2) Actuator Torque Limits · p. 3

3) Actuation-Consistency Constraint:

  • The actuation-consistency constraint bounds desired force components in the SVD null-space basis, reducing reliance on forces invisible to finger-joint torques.The basis is computed from the full SVD of Jact, with rank determined using a singular-value tolerance of 10^-6.Source: 3) Actuation-Consistency Constraint · pp. 3–4
  • At each control cycle, the controller updates the contact set and Jacobians and resolves the QP; if solving fails, it retains the last feasible torque command, initialized to zero.The constraint is a conservative allocation restriction, not a guarantee that physical contacts realize the desired forces; realization also depends on contact compatibility, compliance, and passive reactions.Source: 3) Actuation-Consistency Constraint · p. 4

D. Integration with Reactive Reaching

Reactive reaching is added ahead of force regulation so the system can approach a tracked object, acquire a grasp, and return to reaching if the grasp is lost.

D. Integration with Reactive Reaching

  • Target grasps from Dexonomy are paired with pre-grasps at least 3 cm from the object, and both targets follow its tracked pose.The controller selects the pre-grasp closest to the current palm pose for collision-free approach.Source: D. Integration with Reactive Reaching · p. 4
  • During reaching, a joint-velocity command jointly regulates all five fingertips and the palm under linearized self-, environment-, and hand–object-collision constraints.Fingertip task weights increase near the target while palm weights remain fixed; a configuration regularizer keeps the hand near a nominal posture.Source: D. Integration with Reactive Reaching · p. 4
  • Geometry triggers stage changes: the controller switches from pre-grasp to grasp below 2 cm mean fingertip error, hands over after thumb-plus-two-finger contact, and resumes reaching when the grasp-loss criterion is met.The grasp-loss test is whether the convex hull of the fingertips and palm still intersects the object's bounding box.Source: D. Integration with Reactive Reaching · p. 4

IV. EXPERIMENTS AND RESULTS

The evaluation combines controlled MuJoCo simulations with physical arm–hand experiments. The simulation protocol disables gravity and applies repeated directional loads to assess grasp retention.

IV. EXPERIMENTS AND RESULTS

  • The framework is evaluated in simulation under controlled perturbations and ablations, and on a physical arm–hand system under human disturbances.The stated default settings include µ = 0.7, a 10 N minimum normal force, and a 0.2 N·m per-joint torque limit; the QP uses OSQP.Source: §IV · p. 4

A. Simulation Studies

  • MuJoCo tests use 79 YCB objects with 10 power grasps each and disable gravity to isolate grasp reliability from gravitational loading.A 0.1 mg force is applied for 1 s along each of ±x, ±y, and ±z; a grasp fails if the grasp-loss criterion is met.Source: A. Simulation Studies · p. 4
  • The perturbation study uses five representative objects, while the contact-count study includes all 790 grasps; Fig. 3 and Fig. 4 depict the respective evaluation measures and scopes.Fig. 3 bins success rate by initial ground-truth contact count; Fig. 4 reports robustness across perturbation angles for five objects, with 50 grasps per method.Source: A. Simulation Studies · p. 4, Figure 3 · p. 5, Figure 4 · p. 5

1) Dynamic Perturbation Resistance:

Real-time force reallocation outperforms non-adaptive execution as objects move, and whole-hand contact coverage outperforms fingertip-only control. The contact-count study also reports its largest method gap for grasps with three to five contacts.

1) Dynamic Perturbation Resistance:

  • Whole-hand control achieves 88.6% success at 90◦ versus 72.4% for Fingertip-only, and ours outperforms both non-adaptive baselines at every perturbation angle.The compared baselines allocate forces once at initialization and keep those allocations fixed as the object moves.Source: 1) Dynamic Perturbation Resistance · p. 5
  • With the actuation-consistency constraint, the measured net contact force at the object’s COM is nearly zero in the illustrated three-finger grasp.The visualization is 2 s after initialization; desired-force arrows use twice the length scale of measured-force arrows.Source: Figure 5 · p. 5

2) Impact of Multi-Link Whole-Hand Contacts:

  • Most failures occur during settling: fixed-allocation baselines lose the object as contacts change, whereas online updates let the grasp settle.The baselines allocate forces over the ground-truth initial contacts, while this method uses its own geometric estimate.Source: 2) Impact of Multi-Link Whole-Hand… · p. 5
  • For grasps with N_gt_c = 3–5 contacts, success is 32–56% versus 82–95%, and the gap closes at N_gt_c ≥10.The study bins 790 power grasps by ground-truth initialization contact count; success rises with contact count for all methods.Source: 2) Impact of Multi-Link Whole-Hand… · p. 5

B. Ablation Studies

The ablations examine robustness to object-pose error, the actuation-consistency constraint, and the number of contact points per link. They show a success-versus-compute trade-off as contact count grows and sustained success under tested constant pose offsets.

1) Effect of Actuation-Consistency Constraint:

  • With the actuation-consistency constraint, measured net contact force is nearly zero rather than a 0.428 N resultant, and precision-grasp success rises from 38.5% to 42.1%.Power-grasp success is 100% with or without the constraint; the low absolute precision-grasp rate is attributed to sparse contacts.Source: 1) Effect of Actuation-Consistency… · pp. 5–6

2) Robustness to Object Pose Error:

  • 87–93% success is maintained under the largest synthetic pose offset, while feedforward success drops from 79% to 54%.Among tested thresholds, dthresh = 1 cm achieves 93% success at (30°, 5 mm); the test does not evaluate time-varying noise, latency, or tracking loss.Source: Figure 6 · p. 6, 2) Robustness to Object Pose Error · p. 6

3) Effect of Max Number of Contacts per Link:

C. Real World Experiments

The real-world system uses a tracked object model and synchronized hand configuration to estimate contacts and regulate force without using fingertip tactile sensors. The object model is reconstructed from an RGB-D scan, and the tracker estimates its 6-DoF pose at 30 Hz.

1) Setup:

  • The 27-DoF platform combines a 7-DoF arm with a 20-DoF hand; fingertip tactile sensors are not used for contact estimation.Contact estimation uses K = 5, and the contact-estimation and QP computations run once per tracked pose using the hand configuration recorded at the image timestamp.Source: 1) Setup · p. 6

2) Object Reconstruction and Tracking:

  • A handheld RGB-D scan supplies the object reconstruction, and the modified FoundationPose tracker estimates the object’s 6-DoF pose at 30 Hz.The pipeline uses Point2Pose for camera-pose tracking, 2D Gaussian Splatting for reconstruction, and Poisson surface reconstruction to extract the mesh and precompute its SDF.Source: 2) Object Reconstruction and Tracking · pp. 6–7
  • Execution snapshots show the pipeline combining reactive reaching with whole-hand force regulation across three objects, while Table I summarizes full-pipeline runs and object losses.The supplied table caption defines losses as the hand losing the object and unrecovered losses as workspace exits; it does not provide cell values.Source: Figure 9 · p. 7, Table I · p. 7

3) Results:

In human-perturbation trials, the system usually recovered after losing the object, and it regulated a manually posed grasp with dorsal finger contacts. The discussion presents whole-hand, sensor-free force regulation and identifies tracking, thin-object, and quasi-static-force limitations.

3) Results:

  • In 78 human perturbations over about 200 seconds, the hand lost the object eight times and autonomously regrasps it after six losses.The other two losses were workspace exits; the robot returned to a safe default pose until the object was repositioned and reaching restarted.Source: 3) Results · p. 7
  • A manually posed grasp with dorsal finger contact remained held under a rod push as the estimated little-finger contact shifted with the object.The example uses fingers alternating across the object’s front and back, including contact on the little finger’s dorsal surface.Source: 3) Results · p. 7, Figure 10 · p. 8

V. DISCUSSION AND CONCLUSION

  • The conclusion describes sensor-free whole-hand force regulation using tracked object pose and joint angles, with forces recomputed as contacts change.The discussion also reports simulation gains over non-adaptive and fingertip-only baselines and hardware recovery after most losses.Source: §V · p. 7
  • The controller depends on an object model and pose tracker, and heavy occlusion can make contact estimates fail; thin objects are also difficult under millimeter-scale pose error.The force allocation is quasi-static and does not explicitly compensate for gravity, inertia, or external loads; it also omits passive compliance and contact preloads.Source: §V · pp. 7–8
Loading 2609.30082v1…