PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Gestures: Estimating Full Hand Pose and Contact Forces from Wrist-Worn Pressure Sensor Array

Svetoslav Kolev, Lingni Ma, Michael Goesele, Renzo De Nardi, Jakob Engel, Richard Newcombe

arXiv:2609.16518v1cs.HCcs.RO

TL;DR

Capturing full hand motion and interaction forces remains difficult in everyday settings because vision is occluded and force is not directly visible. The paper uses a wrist-worn capacitive pressure array with a recurrent sequence model to estimate continuous full-hand pose and distributed contact force, finding that external pose conditioning improves force decoding.

  • Problem

    Everyday systems need both full hand motion and interaction force, but vision is vulnerable to occlusion and cannot directly observe force.

  • Method

    A wrist-worn capacitive pressure array feeds a recurrent sequence-to-sequence model that predicts hand pose, interaction force, or both.

  • Results

    External hand-pose cues clearly improve force decoding, whereas joint pose-and-force prediction and self-predicted pose conditioning do not significantly improve force estimation.

  • Takeaways & Limitations

    The wristband can serve as a complementary wearable that supplies force information and maintains hand-state estimation when vision is occluded or out of frame.

  • Takeaways & Limitations

    The pose-conditioned force result uses MoCap-derived pose as a near-oracle, so its approximately 25% MAE reduction is an upper bound for realistic vision-supplied pose.

Abstract

from arXiv · show

Capturing hand motion and interaction forces is critical for interactive computing, VR, and high-fidelity tactile demonstrations for robot learning. We introduce a wrist-worn pressure-sensing wristband that recovers continuous full-hand pose and distributed contact force on a single wearable. The system consists of flexible capacitive sensor arrays around the wrist, which require no electrical skin contact, and a recurrent network that maps the resulting pressure signal to hand state. Our key insight is that muscle contraction and tendon displacement produce pressure patterns, which correlate strongly with hand pose and interaction force. To validate this, we collect synchronized recordings of wrist pressure, optical motion-capture hand pose, and tactile-glove interaction force, covering isolated finger motion, fingertip-force stress tests, and natural hand-object manipulation. On isolated single-user motion the wristband attains $4.6^\circ$ mean finger-joint MAE, and across four users manipulating everyday objects it estimates per-finger contact force at $R^2=0.57$, which an external pose signal brings up to $0.75$. We see the wristband as one node in a constellation of everyday wearables -- e.g. paired with an egocentric camera -- adding the contact force that vision cannot observe and taking over when the hand is occluded.

1 Introduction

The paper addresses the challenge of capturing full hand motion and interaction forces in everyday settings. It proposes a wrist-worn pressure band that maps forearm pressure patterns to hand pose and force, complementing vision during occlusion.

  • 1 Introduction: Vision captures finger motion only when the hand is visible and cannot directly observe force, while gloves can be accurate but obtrusive.Pressure coverage on gloves may also miss hand surfaces involved in everyday object handling.
  • 1 Introduction: Wrist-worn pressure sensing targets both continuous full-hand pose and interaction force from a practical distal-forearm wearable.Forearm muscle and tendon activity produces pressure changes at the band–skin interface, enabling unobtrusive sensing without electrical skin contact.
  • 1 Introduction: The wristband is intended to complement an egocentric camera, providing pose when the hand is occluded and force that vision cannot observe.The authors frame the system as one node in a constellation of everyday wearables rather than a standalone replacement for vision.
  • 1 Introduction: The paper contributes continuous full-hand pose and continuous per-finger and per-taxel force estimation from a single wrist-worn pressure array.It also reports that externally provided hand pose reduces force-estimation MAE, extending pose-conditioned force decoding to pressure sensing.

2 Related Work

Prior wrist-worn systems infer hand state through diverse sensing modalities, but continuous full-hand pose and per-finger force remain comparatively underexplored. This work uses a single capacitive FMG wristband to estimate both outputs, optionally conditioning force on external pose.

  • Sensing rationale: FMG senses mechanical deformation from muscle contraction and tendon displacement without skin-electrode coupling, offering a low-power, low-cost, sweat-robust sensing route.The paper positions FMG as complementary to modalities that measure other parts of hand state.
  • Comparison scope: Table 1 organizes representative wearable forearm and wrist systems, but differing metrics and protocols prevent direct comparison across rows.Per-row operating points and metric definitions are deferred to the supplementary discussion.
  • Pose estimation: Continuous full-hand pose remains difficult for wrist-worn systems, whose pressure-based predecessors recover only partial hand representations.Vision offers strong accuracy but requires instrumented spaces or line-of-sight, while eyes-free alternatives include sEMG, EIT/RF, and acoustics.
  • Force estimation: Per-finger interaction force has generally required dense HD-EMG, ultrasound, or vision, while prior FMG studies targeted aggregate grip or force/torque.These approaches impose instrumentation, complexity, or occlusion constraints that limit practical wearable deployment.
  • This work: The proposed system uses one wrist-worn capacitive array for full-hand pose and per-taxel force, with optional external-pose conditioning for force estimation.This contrasts with systems combining separate sensors or requiring continuous camera line-of-sight.

3 Hand State from Pressure-sensing Wristband

The paper models hand pose and external contact force from pressure measured around the distal forearm, using biomechanical coupling between forearm tissues and hand state. A customized capacitive wristband, synchronized optical and tactile supervision, and a sequence model support this approach.

  • Biomechanical basis: Muscle contraction and tendon displacement alter distal-forearm stiffness and contour, transmitting mechanical changes to the skin that the wristband measures.This biomechanical coupling motivates pressure sensing as a proxy for hand pose and interaction force.
  • Hand-state target: Pressure around the distal forearm is used to infer both kinematic pose and contact force, while internal muscle tension under co-contraction remains outside the paper’s target.The authors distinguish pose, external force, and internal tension as separate components of hand state.
  • Wristband hardware: The wearable uses three or four capacitive arrays in a tapered fabric wristband, providing 540 or 720 total taxels at approximately 50 or 45 Hz.Commercial PPS arrays are packaged in dedicated pockets to maintain sensor placement around different forearm sizes.
  • Ground-truth system: Supervision combines 19-marker OptiTrack hand tracking with a rigid proximal-forearm marker tree and 65-taxel PPS tactile gloves measuring interaction forces.The systems provide hand pose, distal-forearm rotation, and distributed finger-and-palm force measurements.
  • Synchronization: Glove latency and clock drift are corrected by aligning repeated peak-force events and continuously tracking the glove clock during recording.The tactile glove streams at 100 Hz but can accumulate about one second of drift over ten minutes.
  • Sequence model and data: The learning task is sequence-to-sequence prediction, using temporal coherence and separate normalized projections for raw pressure and its temporal derivative.The unified temporal architecture applies GeLU to the raw branch and Tanh to the derivative branch; recordings use separate donning sessions with varying placement and tension.

4 Experiments and Results

The experiments evaluate wristband-based hand-pose and force estimation across datasets, users, sessions, and force regimes. Results show generally accurate pose and force tracking, with external pose improving force decoding while wrist rotation and sparse training coverage remain important error sources.

  • Experimental setup: The evaluation uses per-user models and leave-one-recording-out cross-validation, testing across separate band donning sessions.The experiments report pose on HP and HOM and force on HOM, with the auxiliary FF dataset used as a reference.
  • Hand pose: Predictions are usually close to ground truth, but wrist rotation can produce missed flexions or wrong-finger attributions.Errors are worst where wrist orientations are sparsely sampled during training, requiring interpolation across orientations.
  • Force estimation: External kinematic input improves force decoding, whereas joint multitask prediction and self-predicted pose conditioning do not significantly help.The result supports using independent pose cues to distinguish finger-motion effects from force-production effects.
  • Force error vs. magnitude: Below ∼0.3 N, force estimates approach the y=x error baseline; from ∼0.3 to ∼5 N, MAE grows sub-linearly and pose conditioning helps.Above ∼5 N, curves steepen and inter-user spread widens as high-force samples become sparse.
  • Data scaling: A single ∼20-minute recording yields ∼1.5–1.8× the full-training loss, while two recordings recover most of the gap.On HP, loss reaches within ∼5% of full training after ∼4–5 recordings, although performance continues improving with more data.

5 Discussion

The discussion positions the wristband as a complementary wearable that jointly estimates pose and force from forearm pressure. Its benefits are bounded by biomechanical confounds, reference coverage, oracle-like pose conditioning, and weaker sensitivity to static muscle activity.

  • Discussion: A wrist-worn capacitive pressure array recovers continuous full-hand pose and distributed contact force on one wearable.External pose reduces force MAE by ∼25%, and this gain is not reproduced by multitask prediction or the model’s own pose estimate.
  • Complementary sensing: Wristband pressure complements vision by supplying force and maintaining hand-state estimates when the hand is occluded or out of frame.Vision can provide pose when visible, while the wristband contributes information that vision cannot directly observe.
  • Biomechanical limits: Wrist rotation can swamp subtler finger-flexion and contact-force signatures, especially near sparsely sampled extremes of the wrist range.Longitudinal tendon motion produces little circumferential pressure signature, so transverse and bulk forearm deformation carry much of the usable signal.
  • Evaluation scope: Natural object-manipulation force evaluation is more representative than fixed isometric tests, but its structured contact manifold limits how fully results generalize mechanically.The tactile glove also leaves lateral, dorsal, and webbing surfaces uninstrumented and lacks an independent force reference.
  • Limitations: Pose-conditioned force results use MoCap-derived pose as a near-oracle, so the ∼25% MAE reduction is an upper bound for realistic vision pipelines.The discussion identifies closing this gap under vision-supplied pose as a natural next experiment.
  • Comparison with EMG: FMG is weakest for static poses, isometric holds, and co-contraction because it senses mechanical loading rather than muscle activation.Surface EMG captures these conditions more directly, and EMG–FMG fusion outperforms either modality alone.

A.1 Extended Background

Hand and wrist motion arises from coordinated extrinsic and intrinsic muscle–tendon groups whose differential activation produces characteristic pressure patterns at the band–skin interface.

  • Biomechanical background: Opposing muscle–tendon groups regulate each degree of freedom, producing pressure patterns as their activation changes.The wrist contributes flexion–extension, radial–ulnar deviation, and pronation–supination rotations.

A.1.1 Biomechanical Background.

Everyday hand behavior rarely isolates intrinsic muscles, but combined activation patterns still create measurable mechanical signatures at the distal forearm.

  • Biomechanical background: Combined muscle activation remains measurable at the distal forearm even though everyday hand behavior rarely isolates intrinsic muscles.

A.1.2 Why Pressure? The Surface-Deformation Regime of Wrist

The paper motivates pressure sensing as a direct measure of wrist surface deformation, aiming to capture hand-state information without electrode–skin confounds.

  • Pressure sensing isolates mechanically mediated surface deformation without the electrode–skin interface confound present in EIT and related electrical systems.The authors hypothesize that this mechanical component contains much of the information needed to decode hand state.
  • On a matched participant, anatomy, and protocol, dense pressure sensing outperforms the EIT band on every pose axis.

A.1.3 A History of Wrist- and Forearm-Worn Hand Sensing.

Wrist- and forearm-worn sensing has evolved from pressure-based prosthetic and gesture control toward continuous pose–force estimation across multiple sensing modalities.

  • Force myography: from prosthetic control to continuous sensing: Force myography infers hand state from forearm tissue deformation, progressing from prosthetic finger intent to richer continuous pressure images around the wrist.The lineage includes residual kinetic imaging, GestureWrist, WristFlex, and flexible capacitive arrays.
  • Electromyography: the best-benchmarked competitor: sEMG is the most mature wrist-worn modality and is supported by public benchmarks for gesture recognition and continuous pose.
  • Ultrasound and acoustics: Ultrasound and acoustics measure changing surface geometry, with air-coupled systems trading spatial detail for lighter, contact-free packaging.Reported targets include joint sets or aggregate force rather than per-taxel contact.
  • Vision: Vision can achieve high hand-tracking accuracy but requires an unobstructed optical path or instrumented space, limiting wrist-worn use during occlusion.
  • Combined pose–force estimation and where we sit: Combined pose–force systems use dense EMG, musculoskeletal models, or posture-conditioned force estimation, situating this work among joint kinematic–kinetic approaches.

A.2 Datasets

The study uses three datasets that pair wristband pressure with complementary pose and force measurements to test hand-pose decoding, force estimation, and everyday manipulation.

  • Across all datasets, the wristband is donned and doffed for every recording, with placement and strap tension standardized by inspection.
  • Hand Pose (HP): The HP dataset combines pressure wristband recordings with motion-capture pose while emphasizing combined finger–wrist motion to separate overlapping signal effects.It includes varied flexion, multi-finger combinations, wrist orientations, tapping, and micro-gestures.
  • Fingertip Force (FF): The FF dataset records wristband pressure and tactile-glove force without MoCap, stressing fingertip-force prediction under strong interference from hand and wrist pose.It spans fingertip force, hand pose, and wrist pose, including varied grasping and everyday interactions.
  • Hand-Object Manipulation (HOM): The HOM dataset synchronizes hand pose, wrist pose, tactile force, and wristband pressure during everyday object pick-and-place and manipulation activities.

A.2.2 Auxiliary Fingertip-Force (FF) Dataset and Results.

The auxiliary FF dataset provides a MoCap-free stress test for fingertip-force estimation, with representative recordings and preprocessing that align multimodal streams at 50 Hz.

  • The three datasets provide representative synchronized samples linking wristband pressure maps with pose, tactile readings, and RGB scenes.
  • The FF examples pair RGB scenes, tactile-glove taxel layouts, and wristband pressure maps while varying grasp combinations, grip forces, and everyday object interactions.
  • The FF dataset is a single-participant, MoCap-free fingertip-force stress test used as auxiliary training data and in ablations rather than as a headline main-paper result.Its standalone force-estimation results are reported separately.
  • Preprocessing calibrates and masks tactile taxels, smooths streams with zero-phase exponential filtering, and resamples them onto a common 50 Hz timeline.The smoothing bandwidth is controlled by tau, with tau set from the 50 Hz target before linear interpolation.

A.3 Model and Training Details

The model trains on fixed-length pressure sequences and applies supervision only after recurrent warm-up. Overlapping inference windows are averaged per timestamp before evaluation.

  • N=100 samples provide 2 seconds of context at 50 Hz, while batch size 128 processes 256 seconds of context per optimization step.Training runs for about 80 iterations per epoch and roughly 600 effective epochs over 50,000 iterations.
  • Supervision uses only the final N/2 outputs, giving the recurrent state warm-up context before predictions are scored.The hidden state starts at zero, and training and testing use the same context length.
  • Each length-N window predicts its final N/2 steps, and overlapping stride-1 predictions are averaged for every timestamp.A timestamp can receive up to N/2 valid predictions before metrics are computed.

A.4 Metrics

The evaluation reports scalar errors in task-appropriate units and uses pose metrics derived from forward kinematics. MPJPE is reported in finger-only and with-wrist variants to separate finger articulation from full end-to-end wrist error.

  • Scalar quantities use MAE and R^2, with joint angles in degrees, force in Newtons, and taxel pressure in PSI.R^2=1 denotes perfect prediction, R^2=0 matches predicting the ground-truth mean, and negative values are worse than mean prediction.
  • MPJPE measures mean Euclidean 3D skeleton-joint distance, with finger-only isolating articulation and with-wrist capturing end-to-end wrist error.Finger-only evaluates 16 joints using ground-truth wrist orientation, whereas with-wrist evaluates 20 finger skeleton points using predicted wrist orientation.
  • Fingertip position error and MPJPE are reported in millimetres after forward kinematics converts predicted joint angles into skeleton positions.

A.4.2 Pose Position Metrics.

The detailed ablations show that temporal context, input richness, and recurrent capacity shape pose accuracy, with wrist-side pressure regions carrying the most useful signal. Performance saturates with moderate context and model size, while errors increase at sparse wrist-angle coverage and extreme spatial subsampling.

  • A.5 Detailed Results: Finger error is lowest near the center of each wrist-angle distribution and rises toward sparsely sampled extremes, especially for radial–ulnar deviation.This dependence reflects uneven coverage of wrist configurations rather than a uniform pose-estimation error.
  • A.6 Model Capacity Ablation: A 256-unit recurrent layer removes roughly twice as much HP error as an equally scaled input-reduction layer, making recurrent capacity the more actionable allocation.The (64, 256) configuration stays within 1–2 percentage points of (256, 256) across datasets while using fewer parameters.
  • A.7 Context Length Ablation: A 100-sample context captures nearly all recoverable temporal benefit across datasets, while extending it to 200 samples reduces error by less than ∼2%.The same saturation pattern appears for joint angles, fingertip force, and HOM force outputs.
  • A.7 Context Length Ablation: Removing pose input makes longer context more valuable: P1 improves by ∼13% without pose versus ∼7% with pose from blk-16 to blk-100.With externally supplied pose, force-prediction curves are nearly flat across context lengths; without it, longer history partly compensates for missing kinematic cues.
  • A.8.2 Where on the forearm does the window matter?: Finger-joint and fingertip-force tasks favor windows centered toward the wrist side, consistent with more discriminative distal-forearm pressure signals near the wrist.The distal half is described as the more information-dense region for hand-state decoding, while very short windows are typically 10–30% worse than the full-strip baseline.
  • A.8 Spatial Subsampling: Extreme column subsampling adds 6–16% error on HP/wrist and FF, whereas moderate column reduction remains comparatively inexpensive for kinematics.The contrast is attributed partly to contiguous row removal excising an anatomical band, while strided column removal retains redundant neighboring samples.
Loading 2609.16518v1…