Source-linked AI summary
Wearable Multimodal Human-Machine Interface for Integrated Hand Intentions Decoding in Dynamic Teleoperation
Jiaxuan Li, Yinshi Wu, Xiao Zhang, Hongyu Wang, Renzhen Le, Zhenzhi Ying, Liming Shu
TL;DR
Existing teleoperation interfaces face trade-offs in decoding multiple hand intentions while remaining wearable, and unrestricted wrist and forearm motion complicates stable decoding. The paper presents a wearable sEMG–IMU interface and unified framework for simultaneous pose, gesture, and force decoding, reporting gesture and force results alongside simultaneous estimation of all three intentions.
Problem
Existing interfaces trade off the diversity of decoded hand intentions and usability, while stable gesture decoding under practical wrist and forearm motion remains a challenge.
Source: §I · p. 1Method
The MI-DHMI combines high-throughput sEMG with hand- and forearm-mounted IMUs and a unified framework for simultaneous hand-pose, gesture, and force decoding.
Source: §I · p. 2, §VI · p. 10Results
The system simultaneously estimates gesture, force, and pose; attention-based gesture decoding reached 97.16%, and MI-DHMI sEMG force regression achieved R^2 values of 0.87–0.95.
Source: A. Gesture Decoding · p. 7, B. Force Regression · p. 7, §V · p. 9Takeaways & Limitations
Operating without external cameras, markers, or environmental infrastructure, MI-DHMI is suited to teleoperation scenarios requiring mobility, portability, and consistent performance.
Source: §V · p. 9
Abstract
from arXiv · showhide
Under ubiquitous teleoperation environments with optically challenging conditions, an interface for tele-operated grasping that combines wearability with precise decoding of hand intentions (hand pose, gestures, and grasping force) is essential. Yet, existing interfaces often fall short in meeting these demands, compromising either the diversity of multiple intentions decoding or wearability. To address this, we developed a novel Multiple Intentions Decoding Human-Machine Interface (MI-DHMI) that integrates high-throughput surface electromyography (sEMG) sensors with hand-mounted and forearm-mounted inertial measurement units (IMUs). The developed interface is supported by a unified framework for simultaneous multiple intentions decoding. By employing multimodal deep learning and hardware design with a low noise floor, the decoding framework selectively focuses on the sEMG components that are genuinely associated with finger movements. This effectively reduces decoding errors caused by sEMG variability during unconstrained upper-limb motions, thereby significantly enhancing robustness. Even under unconstrained wrist and forearm motion, the interface achieves a gesture recognition accuracy exceeding 97%, grasping force estimation with $R^2 = 0.95$, and hand pose decoding consistent with the actual hand pose, outperforming baseline devices and algorithms. Ablation studies further validate the effectiveness of the proposed decoding framework. Finally, two online experiments were conducted to validate the device, demonstrating its superior performance in high-stability tasks, including a pouring task and object grasping. The developed interface provides a new solution of a fully wearable, multiple intentions decoding system, offering effective support for ubiquitous teleoperation and contributing to the advancement of human-machine interaction research.
I. INTRODUCTION
The paper targets fully wearable teleoperation interfaces that can decode multiple hand intentions during unrestricted movement, despite existing trade-offs between intention diversity and usability. It introduces the MI-DHMI and a unified framework for simultaneous hand-pose, gesture, and force decoding.
I. INTRODUCTION
- Existing interfaces trade off intention diversity and usability, while coupled wrist and forearm motion makes stable gesture and force decoding in wearable teleoperation an open challenge.Joysticks cannot decode gestures, optical systems depend on external infrastructure and generally cannot capture force, and gloves can constrain mobility or tactile perception.Source: §I · p. 1
- The MI-DHMI integrates high-throughput sEMG acquisition with hand- and forearm-mounted IMUs to capture hand–forearm kinematics and sEMG while preserving wearability.The work describes this as the first fully wearable multimodal HMI for simultaneously decoding hand poses, gestures, and grasping force under unconstrained wrist and forearm motion.Source: §I · pp. 1–2
- The unified framework combines ADF-Net, MCF-Net, and SZPR to infer hand gestures, force, and poses simultaneously under unconstrained wrist and forearm motion.The paper reports validating the interface through ablation studies and online teleoperation experiments.Source: §I · p. 2
A. System Overview
- The system pairs a forearm armband containing sEMG and an IMU with a hand-mounted IMU, using the two IMUs’ orientation information to derive wrist angles for gesture and force decoding.The overview states that data are transmitted wirelessly to a central computing unit over a local area network.Source: A. System Overview · p. 2
B. Notation and Coordinate Systems
This section establishes the paper’s notation and coordinate frames, describes the wearable sensing hardware, and outlines hand-pose reconstruction with orientation calibration and Soft ZUPT-based position estimation.
B. Notation and Coordinate Systems
- Scalars, vectors, matrices, estimated values, and time-varying vectors receive distinct notation, while the world frame is defined with its z-axis opposite gravity and quaternion multiplication is denoted by ⊗.The paper omits the world-frame superscript on vectors for simplicity.Source: B. Notation and Coordinate Systems · p. 2
C. Hardware Architecture of the MI-DHMI
- The armband uses a 64-channel dry-electrode array sampled at 2 kHz and an IMU sampled at 400 Hz, alongside a 400 Hz hand-mounted IMU; the complete interface weighs approximately 161 g.The array is arranged in a 4×16 grid, with 12 mm electrode spacing along the forearm axis; the hand IMU is mounted at the dorsum’s center.Source: C. Hardware Architecture of the MI-DHMI · pp. 2–3
D. Hand Pose Estimation in the Decoding Framework
- For hand-pose estimation, SZPR represents position and orientation, and a constant calibration quaternion compensates for fixed misalignment between the IMU and paper world frames.The position-estimation procedure takes hand-frame acceleration and angular velocity, estimated orientation, sampling interval, initial velocity and position, Soft ZUPT gain, and motion state, and returns estimated hand position.Source: D. Hand Pose Estimation in the… · pp. 3–4
- Hand-frame acceleration is rotated into the world frame and integrated to obtain velocity and position; Soft ZUPT adapts velocity attenuation using a motion-state indicator to mitigate drift.A lightweight MLP estimates the motion state from sliding-window acceleration and angular-velocity measurements; the gain increases when stationary and decreases when moving.Source: D. Hand Pose Estimation in the… · pp. 3–4
E. Gesture Recognition in the Decoding Framework
Gesture recognition combines sEMG with hand–forearm IMU information to address motion-related sEMG disturbances. ADF-Net uses data-level and feature-level fusion, with learned temporal features and attention.
E. Gesture Recognition in the Decoding Framework
- Hand IMU quaternions are expressed in the forearm frame to represent wrist angles and compensate for disturbances induced by unconstrained wrist and forearm motion.The relative hand orientation is computed by initialization-based quaternion calibration, with the wrist held neutral and stationary at initialization.Source: E. Gesture Recognition in the Decoding… · p. 4
- ADF-Net integrates complementary sEMG and IMU information through both data-level and feature-level fusion for gesture recognition.Its architecture includes a feature-extraction module and a lightweight attention mask module.Source: E. Gesture Recognition in the Decoding… · p. 4, Figure 4 · p. 4
- The feature-extraction backbone is a 1D-CNN that processes channels along the temporal dimension, with residual connections intended to improve generalization while preserving input dimensions.Source: E. Gesture Recognition in the Decoding… · p. 4
1) Data-level fusion:
Data-level fusion concatenates sEMG and hand–forearm quaternion inputs while preserving temporal length and channel dimensionality.
1) Data-level fusion:
- The data-level branch concatenates the sEMG time-window matrix and hand–forearm quaternion matrix along the channel dimension before feature extraction.Source: 1) Data-level fusion · p. 4
- Preserving temporal length and channel dimensionality enables the network to learn coupled representations of sEMG activity and wrist kinematics.Source: 1) Data-level fusion · p. 4
2) Feature-level fusion:
Feature-level fusion extracts representations from each modality independently, then concatenates them for classification.
2) Feature-level fusion:
- The feature-level branch first extracts features independently from each modality.Source: 2) Feature-level fusion · p. 5
- It concatenates the modality features for classification, a fusion intended to mitigate insufficient feature capture from modality heterogeneity and improve decoding robustness.Source: 2) Feature-level fusion · p. 5
3) Attention mask module:
The attention-mask and classification modules combine learned features, while the force-regression pathway uses sEMG and hand–forearm quaternion information to estimate grasp force under motion.
3) Attention mask module:
- The attention-mask module generates channel-wise weights from pooled temporal features and applies them to the input to produce a weighted feature representation.Source: 3) Attention mask module · p. 5
4) Classification module:
- The classification module combines data-level and feature-level branch outputs with learned weights, then uses temporal average pooling and a two-layer fully connected network for classification.Source: 4) Classification module · p. 5
F. Force Regression in the Decoding Framework
- The force-regression MCF-Net integrates time-windowed sEMG samples with hand–forearm quaternion windows to compensate for sEMG fluctuations induced by unconstrained wrist motion.It applies temporal attention to modality features, concatenates the enhanced features, and uses temporal average pooling followed by an LSTM to estimate grasp force.Source: F. Force Regression in the Decoding… · p. 5
A. Data Acquisition
The experiments collected synchronized hand-intention signals during freely moving gestures, then compared decoding modalities and fusion strategies before testing teleoperation tasks.
A. Data Acquisition
- Ten participants performed seven grasping gestures with freely moving arms and wrists while sEMG, force, IMU, and ground-truth hand-pose data were recorded.Data were split by session into training, validation, and test sets at a 3:1:1 ratio.Source: A. Data Acquisition · pp. 5–6
B. Ablation and Comparative Experiments
- Comparative experiments evaluated gesture and force decoding against MYO, tested modality contributions and attention modules, and compared data-, feature-, and hybrid-fusion strategies.Classical gesture baselines used per-channel sEMG mean absolute value features; gesture accuracy and force R2 were the evaluation measures.Source: B. Ablation and Comparative Experiments · p. 6
C. Teleoperation Experiments
- Teleoperation trials tested sequential pouring and object-placement tasks, mapping human hand position and orientation to the robot while decoded gestures and force controlled grasping.Gesture transitions required the same prediction in three consecutive time windows.Source: C. Teleoperation Experiments · p. 6
IV. RESULTS
Gesture decoding benefited from attention and multimodal fusion, while attention-based multimodal decoding also yielded strong force estimates. Feature visualizations showed better separation with multimodal fusion than with either signal alone.
A. Gesture Decoding
- 97.16% gesture accuracy was achieved by combining data- and feature-level fusion with attention, the highest reported result among the compared strategies.The comparison included single-modality sEMG, data-level fusion, feature-level fusion, and their combination.Source: Figure 7 · p. 7, A. Gesture Decoding · p. 7
- Attention-based multimodal fusion produced well-separated, compact gesture-feature clusters, while single-modality features had limited separability under unconstrained wrist motion.The reported visualization attributes complementary discrimination of challenging gestures to IMU signals alongside sEMG.Source: A. Gesture Decoding · p. 7
B. Force Regression
- 0.94–0.95 R2 for MI-DHMI and 0.90–0.95 R2 for MYO were achieved with attention-based multimodal force decoding.Single-modality sEMG yielded R2 values of 0.87–0.95 for MI-DHMI and 0.78–0.92 for MYO.Source: B. Force Regression · p. 7
C. Hand Pose Estimation
The pose and teleoperation evaluations report IMU tracking behavior and stable force-regulated grasping; the discussion describes multimodal fusion and full wearability as advantages while identifying open deployment questions.
C. Hand Pose Estimation
- Position decoding drifted over time, whereas orientation error was relatively small and compensated during movement; Fig. 10 compares predicted and captured hand poses.Reconstructed trajectories were compared with ground-truth trajectories.Source: C. Hand Pose Estimation · p. 7, Figure 10 · p. 8
D. Teleoperation Experiments
- The pouring evaluation recorded hand and robotic positions and orientations, sEMG signals, and grasping force; sEMG energy was calculated as windowed channel RMS averaged across channels.Fig. 11 lays out the corresponding human, estimated, and robotic signals and their errors.Source: D. Teleoperation Experiments · p. 7, Figure 11 · p. 9
- During pouring, IMU trajectories generally followed motion-capture positions, robotic motion reached near-millimeter positional accuracy after alignment, and force control supported stable grasping.Human-in-the-loop adjustments partially compensated for trajectory deviations; predicted force regulation avoided unintended opening, slippage, or excessive or insufficient grasp force.Source: D. Teleoperation Experiments · pp. 7–8
- LP gestures had comparatively lower online grasping success rates, primarily because they required greater positional accuracy during grasp execution.Each gesture class was tested in ten repeated grasping trials.Source: D. Teleoperation Experiments · p. 8
V. DISCUSSION
- MI-DHMI simultaneously estimates gesture, force, and pose without external cameras, markers, or environmental infrastructure.The paper describes it as the first fully wearable multimodal HMI for ubiquitous teleoperated grasping under unconstrained wrist and forearm motion.Source: §V · p. 9
- Multimodal fusion combines wrist kinematics and muscle activity to improve feature separability and robustness in gesture and force decoding; Figs. 8 and 9 visualize feature distributions and force-regression comparisons.The discussion attributes reduced rotation-related interference and better handling of motion-induced sEMG variability to joint modeling.Source: Figure 8 · p. 8, Figure 9 · p. 8, §V · p. 9
- Cross-subject generalization remains to be explored, and future work includes evaluating latency, power consumption, and long-term stability in real-world teleoperation.The authors also propose subject-independent and adaptive training strategies and further study of forearm IMUs and adaptive fusion.Source: §V · pp. 9–10
VI. CONCLUSION
The conclusion presents MI-DHMI as a fully wearable platform for decoding multiple hand intentions during unconstrained motion. Its comparative experiments and teleoperation tests support its practical feasibility for ubiquitous teleoperation.
VI. CONCLUSION
- MI-DHMI combines high-throughput sEMG and hand- and forearm-mounted IMUs to decode pose, gestures, and force during unconstrained motion; pouring and multi-gesture tests demonstrated practical feasibility.The conclusion also reports that ablation and comparative experiments showed advantages from the hardware design, multimodal fusion, and attention modules.Source: §VI · p. 10