Capturing complex hand movements and object interactions using machine learning-powered stretchable smart textile gloves

1 Introduction

• 1.21 and 1.45 degrees average joint angle estimation RMSE were reported for intra- and inter-subject cross-validation, respectively, exceeding published wearable-device and computer-vision systems.The authors also compare the system's accuracy with costly motion-capture systems, without their field-of-view limitations.Source: §1 · p. 5
• The gloves track complex hand and finger movements during object interactions, including scenarios where objects or fingers occlude camera views.The authors also report that the system is not subject to camera field-of-view limits, long setup times, and sensitivity to occlusion and image noise.Source: §1 · p. 5
• Data augmentation improves robustness by about two times amid variations in sensor performance and fit to the subject.The technique targets variations in the sensors and their fit to users.Source: §1 · p. 5
• Typing accuracy on a mock paper keyboard reached 97.80%.The paper also demonstrates dynamic gesture recognition, static gesture recognition, and object identification.Source: §1 · p. 5
• Dynamic gesture recognition achieved 94.05% inter-subject and 97.31% intra-subject cross-validation accuracy for 50 gestures; static gesture recognition achieved 94.60% and 97.81% inter- and intra-subject cross-validation accuracy, respectively.Source: §1 · p. 5
• Object recognition from grasp patterns reached 90.20% inter-subject and 95.02% intra-subject accuracy for 34 objects.The authors describe the glove as a tool for dexterous hand and finger movements beyond conventional gesture recognition.Source: §1 · p. 5

2 Smart Textile Gloves

• The glove places HSYs at finger joints and tips, the palm, and thumb joints, while two 9-axis IMUs track wrist orientation.The yarns detect local fabric stretches and pressures from finger movement and object interaction; the IMUs estimate wrist quaternion angles.Source: §2 · p. 6
• A wireless processing board transmits sensor data to an external receiver, where a machine-learning pipeline supports joint-angle estimation, visualization, contact detection, and gesture and object detection.GlovePoseML estimates hand joint angles and tactile information from incoming data.Source: §2 · p. 7

3 Helical Sensor Yarns

• HSYs with 120–200 µm PDMS thickness and 40–60 µm core HSY provide optimum performance, and HSYs can reach up to 155% stretchability.Source: §3 · p. 8
• Helical sensor yarns respond to strains from 0.005% to 155%, covering both subtle and large deformations.The sensors also respond to pressure from 1.7 kPa to 1.2 MPa.Source: §3 · p. 8
• The HSYs show less than 10% sensitivity degradation when immersed in water and stability during laundry cycles.The glove system was tested for more than tens of washing cycles after removal of the rechargeable battery.Source: §3 · pp. 9–10

4 Machine Learning Model

• Training used over 3,000,000 frames sampled at 20 Hz from five participants with different hand sizes performing varied hand movements.The glove carried markers to provide motion-capture ground truth.Source: §4 · p. 10
• GlovePoseML uses a two-layer bidirectional LSTM followed by two fully connected layers to estimate joint angles and tactile information.It processes a two-second history of normalized sensor and motion-capture data.Source: §4 · p. 11
• The model demonstrates high joint-angle accuracy across inter- and intra-subject cross-validation, with results compared against a motion-capture gold standard.The paper reports joint-angle R2 and RMSE results for each joint.Source: §4 · p. 11
• The system operates in scenarios where camera-based methods and wearable devices provide limited or low-accuracy results, including complex movements, visual occlusion, and dark conditions.The authors attribute performance to sensor stability and range, reliable sensor placement, and the machine-learning algorithm.Source: §4 · pp. 11–12

5 Data Augmentation Technique

• Augmentation targets small variations in sensor properties, placement, and fit across subjects or sessions.The paper describes these variations as practical factors affecting the sensors and glove fit.Source: §5 · p. 12
• Pretraining applies random sensor amplitude scaling or DC shift, sensor masking, and additive noise to duplicated training data.The authors say this improves robustness and reduces the need for excessive calibration and retraining when sensor performance or fit varies.Source: §5 · pp. 12–13
• About 2× reduction in average RMSE was reported with augmentation in the presence of unavoidable variations.The comparison is between the original model and one incorporating the data-augmentation technique.Source: §5 · p. 12

6 Application Examples

• The system tracks finger and wrist joints during dynamic movements even in darkness or when an object occludes the motion-capture cameras.Source: §6 · pp. 13–14
• Typing prediction averaged 97.80% accuracy, supporting complex keyboard functions such as holding shift for capital letters.Source: §6 · p. 14
• The examples also include drawing in the air using wrist movements and selecting colors by pinching the thumb with different fingers.Source: §6 · p. 14
• Wrist-angle errors versus motion capture were 2.48° for flexion or extension, 2.34° for abduction or adduction, and 2.54° for supination and pronation.Source: §6 · p. 14
• Dynamic gesture recognition reached 94.05% inter- and 97.31% intra-subject accuracy for 50 gestures; static recognition reached 94.60% and 97.81%, respectively, for 48 gestures.Source: §6 · p. 15
• Object recognition from grasp patterns reached 90.20% inter- and 95.02% intra-subject cross-validation accuracy across 34 objects.Source: §6 · p. 15

7 Conclusion

• The sensor yarns respond to strains from 0.005% to 155% and remain stable during extensive use and washing cycles.Source: §7 · p. 16
• Average joint-angle RMSE was 1.21° for intra-subject and 1.45° for inter-subject cross-validation against motion capture.Source: §7 · p. 16
• Data augmentation enhances robustness to sensor-performance variation, noise, and differences in fit to users.Source: §7 · p. 16
• Demonstrations include real-time movement tracking, recognition of 48 static and 50 dynamic gestures, typing on a mock keyboard, and object recognition from grasp pose and forces.Source: §7 · pp. 16–17

8.1 Fabrication

• HSYs were made by helically electrospinning nanofibers onto spandex yarn, gold-coating them, and placing sensors at designated locations on stretch-knit glove fabric.The glove fabric was a stretchable single-jersey plated weft knit.Source: §8.1 · pp. 17–18

8.2 Characterization of the HSYs

• HSYs were characterized through tensile tests above 2% strain, low-strain tests below 2%, compression tapping, and monitoring after machine washing and drying.Electrical and mechanical properties were monitored after each laundry test.Source: §8.2 · p. 19

8.3 Electronic Hardware and Software

• A custom PCB sampled 25 HSYs and two IMUs at 20 Hz with 12-bit resolution, while software corrected baseline resistance using a 400-sample window.The system used Bluetooth Low Energy connectivity.Source: §8.3 · pp. 19–20

8.4 Data Collection and Training

• Data were collected from five healthy right-handed participants on both hands for joint tracking or typing, gesture recognition, and object grasping.The same stretchable glove pair was used across participants.Source: §8.4 · p. 20

8.4.1 Articulated Dynamic Hand Tracking

• A Bi-LSTM regression model mapped two-second windows of glove sensor data to hand-joint angles, using fused HSY and downsampled IMU inputs.The tracking dataset included over 3,000,000 frames.Source: §8.4.1 · p. 21
• Processing sensor modalities in separate network branches did not significantly improve performance over fusion.The fused input combined HSY and IMU data because of their correlation with joint angles and computational efficiency.Source: §8.4.1 · p. 21
• Data augmentation masked one to three channels, added Gaussian noise to channels, or scaled channels by random factors, followed by multi-task training.The added noise had mean 0 and standard deviation 0.06; scaling factors ranged from 0.5 to 1.5.Source: §8.4.1 · pp. 21–22
• A supplementary video demonstrates articulated dynamic tracking of finger movements.Source: §8.4.1 · p. 22

8.4.2 Keyboard Typing Detection

• Typing detection combined fingertip HSY click sensing with two fully connected layers mapping GlovePoseML outputs to ten keys, trained on 100-character paragraphs and evaluated by five-fold cross-validation.Source: §8.4.2 · p. 22

8.4.3 3D Drawing

• The 3D-drawing app positioned its cursor from the relative quaternion angle of two IMUs and selected colors through thumb pinches detected by fingertip HSY touch sensing.Source: §8.4.3 · p. 23

8.4.4 Dynamic and Static Hand Gesture Recognition

• Gesture recognition used videos of one gesture and a rest position, with 5-second gesture and rest intervals; training updated the final two layers while freezing GlovePoseML.The study collected 20 data sets for each gesture.Source: §8.4.4 · p. 23

8.4.5 Object Detection From the Grasp

• Grasp-based object detection mapped sensor-window tensors labeled with grasped objects through the earlier joint-angle model and two fully connected layers, updating only the final two layers.Source: §8.4.5 · pp. 23–24

Code Availability

• Code supporting the study’s findings is available at github.com/arvintashakori/SmartTextileGlove.Source: Code Availability · p. 24

• AT and PS developed the system model and learning algorithm, alongside the mobile app, data pipeline, acquisition software, Unity application, and firmware.Source: Author Contributions Statement · p. 25
• ZJ and SS designed the yarn-based strain sensors, while a broader team developed the hardware and fabricated the gloves and sensors.Source: Author Contributions Statement · p. 25
• AT performed the experiments and analysis with input from others; ZJW oversaw the project, and all authors contributed to writing.Source: Author Contributions Statement · p. 25
• CN helped fabricate the PCB box and draw the sensor schematic.Source: Author Contributions Statement · p. 25

• The authors report a patent filing based on the work; the remaining authors declare no competing interests.Source: Competing interests · p. 25
• The figure descriptions include testing sensor durability through continuous stretch–release cycles and laundry, washing, and drying cycles.Source: Competing interests · p. 28
• The sensor yarns show sensitivity down to 0.005% strain and are described as outperforming other published wearable sensors; the fabric’s maximum stretch is below 120%.Source: Competing interests · p. 32
• Dynamic gesture recognition reached 97.31% intra-subject and 94.05% inter-subject accuracy, while static gesture recognition reached 97.81% and 94.60%, respectively.Source: Figure 4 · p. 35, Figure 5 · p. 36
• Object detection accuracy was 95.02% in intra-subject and 90.20% in inter-subject cross-validation.Source: Figure 6 · p. 37

gorithm

• The augmentation algorithm takes labeled data, transformation functions, and epochs, then trains a multitask model on original and transformed signals with labels and transformation indicators.The output is a trained multitask model.Source: gorithm · p. 38

algorithm

• Tap detection represents each fingertip’s tap status as a binary function and compares sensor changes with a predefined threshold.The method examines the last four samples to stabilize rising-edge detection.Source: algorithm · p. 40
• The drawing app selects colors when the thumb touches the other fingers, using the tap-detection cost function.The described protocol follows the tap-detection algorithm.Source: algorithm · p. 40

ture recognition - Part 1

• Inter-session confusion-matrix accuracies for subjects 1–5 are 97.82%, 96.61%, 96.78%, 97.83%, and 97.51%.These are the subject-level accuracies reported in Figures 1–3.Source: Figure 1 · p. 41, Figure 2 · p. 42, Figure 3 · p. 43

recognition - Part 1

• Inter-session confusion-matrix accuracies for subjects 1–5 are 98.11%, 98.30%, 97.91%, 98.01%, and 96.70%.These are the subject-level accuracies reported in Figures 4–6.Source: Figure 4 · p. 44, Figure 5 · p. 45, Figure 6 · p. 46

nition - Part 1

• Inter-session confusion-matrix accuracies for subjects 1–5 are 95.96%, 94.38%, 94.74%, 94.17%, and 95.83%.These are the subject-level accuracies reported in Figures 7–9.Source: Figure 7 · p. 47, Figure 8 · p. 48, Figure 9 · p. 49

network branches for different modalities.

• Figure 10 compares the proposed GlovePoseML network with a branched design separating helical sensor-yarn and relative-IMU-angle data, using joint-wise RMSE and R2.Source: Figure 10 · p. 50

parison of using different models.

• Figures 11–12 compare models with corresponding parameters fixed and assess performance under Gaussian noise, sensor masking, and random sensor scaling.The figure captions identify average RMSE or MSE and R2 as the comparison measures.Source: Figure 11 · p. 51, Figure 12 · p. 52

Supplementary Video 1

• The videos demonstrate articulated tracking, mock-keyboard typing, 3D drawing in air, static and dynamic gesture recognition, and grasp-pattern-based object detection.Source: Supplementary Video 1 · p. 54, Supplementary Video 2 · p. 54, Supplementary Video 3 · p. 54