Source-linked AI summary
Exploiting Recurrent Neural Networks and Leap Motion Controller for Sign Language and Semaphoric Gesture Recognition
Danilo Avola, Marco Bernardi, Luigi Cinque, Gian Luca Foresti, Cristiano Massaroni
TL;DR
The paper addresses hand-gesture recognition for communication, particularly sign language, where gestures form time-varying feature sequences. It combines LMC-derived hand features with a DLSTM and evaluates the approach on ASL and SHREC data, reporting high recognition accuracy and comparative superiority.
Problem
Hand-gesture recognition must distinguish numerous static and dynamic gestures represented as time-varying sequences, including complex ASL vocabularies.
Method
The method feeds LMC-derived joint angles, fingertip positions, and other hand features into a DLSTM that models gesture sequences.
Results
The approach recognizes a large ASL gesture set and is reported to achieve high accuracy, including superiority over compared state-of-the-art methods on SHREC.
Takeaways & Limitations
Joint-angle features combined with DLSTM modeling provide a basis for recognizing static and dynamic language gestures across substantial gesture sets.
Abstract
from arXiv · showhide
In human interactions, hands are a powerful way of expressing information that, in some cases, can be used as a valid substitute for voice, as it happens in Sign Language. Hand gesture recognition has always been an interesting topic in the areas of computer vision and multimedia. These gestures can be represented as sets of feature vectors that change over time. Recurrent Neural Networks (RNNs) are suited to analyse this type of sets thanks to their ability to model the long term contextual information of temporal sequences. In this paper, a RNN is trained by using as features the angles formed by the finger bones of human hands. The selected features, acquired by a Leap Motion Controller (LMC) sensor, have been chosen because the majority of human gestures produce joint movements that generate truly characteristic corners. A challenging subset composed by a large number of gestures defined by the American Sign Language (ASL) is used to test the proposed solution and the effectiveness of the selected angles. Moreover, the proposed method has been compared to other state of the art works on the SHREC dataset, thus demonstrating its superiority in hand gesture recognition accuracy.
I. INTRODUCTION
The paper targets hand-gesture recognition for sign and semaphoric communication using discriminative joint-angle features and recurrent modeling of gesture sequences. It evaluates a DLSTM-based approach on numerous ASL gestures and claims high comparative accuracy.
- Hand gestures support communication in sign languages, machine interaction, and other applications, motivating automated recognition.
- The authors seek robust recognition of similar gestures and the highest accuracy compared with current literature.
- The proposed method uses stacked LSTM RNNs to model long-term temporal context and learn across multiple time scales.Two or more stacked LSTM RNNs form a Deep LSTM (DLSTM).
- Joint angles are selected as a simple feature set intended to discriminate hand gestures, especially language gestures.
- The study analyzes a large number of static and dynamic ASL gestures as prerequisites for a wider sign-language recognition system.
- The DLSTM combined with an LMC-extracted hand skeleton is presented as a previously unused configuration for hand-gesture recognition.The LMC was also used to design and collect a high-gesture-count dataset with precise joint-position estimates.
II. RELATED WORK
The paper situates hand-gesture recognition among sensor-based, machine-learning classification methods and presents a DLSTM pipeline that maps temporal feature sequences to class probabilities.
- Existing hand-gesture systems commonly acquire features from sensors and classify them using methods such as SVMs, HMMs, HCRFs, or CNNs.
- Each acquired gesture is represented as a sequence X of feature vectors sampled over at most T time instants by a Leap Motion Controller.
- A DLSTM converts the feature sequence into output probability vectors for each time instant and classifies gestures through a softmax layer over K ASL classes.
- The method’s architecture uses two stacked LSTM RNNs during training and returns class probabilities at each time instant.
A. Feature Extraction
Feature vectors combine hand-joint angles with 3D motion and fingertip information so the model can represent static poses and dynamic gestures. Intra-finger angles address static gestures distinguished by finger spacing.
- Each pose is represented primarily through internal joint angles computed at distal, intermediate, and proximal finger joints.The thumb uses corresponding phalanx or metacarpal segments for its angle definitions.
- 3D palm displacements represent hand translation, while fingertip displacements represent hand rotation in 3D space.
- Intra-finger angles γ1, γ2, and γ3 capture static gestures that differ only in the angles between consecutive fingers.
- The DLSTM input vector concatenates joint angles, palm and fingertip displacements, and intra-finger angles.The listed vector is xt = {ω0, ..., ω4, β0, ..., β4, u0, v0, z0, ..., u5, v5, z5, γ1, γ2, γ3}.
B. Sampling Process
The sampling process selects informative time instants from gesture sequences and produces a fixed number T of feature vectors despite differences in execution speed.
- B. Sampling Process: The procedure identifies significant instants where joint angles or hand-centre displacement vary substantially between consecutive times.The feature signals are smoothed with a Savitzky-Golay filter before significant variations are determined.
- B. Sampling Process: When fewer than T significant instants exist, the remaining samples are selected randomly from the gesture interval.
- B. Sampling Process: When more than T significant instants exist, samples are allocated among features in proportion to each feature’s extrema-based candidate set.The allocation satisfies Σ_g T_g = T.
- B. Sampling Process: After sampling, every acquisition is represented by a sequence of T feature vectors, with selection dynamically determined by feature values.
C. Deep Last Short Term Memory network
The proposed classifier uses stacked LSTM layers to model temporal gesture sequences, preserving information through recurrent memory and gated updates. Its outputs form class probabilities that determine the recognized gesture.
- C. Deep Last Short Term Memory network: The network is composed of multiple LSTM units that address training difficulties associated with ordinary recurrent activation functions.LSTM units use input, forget, and output gates with a recurrently maintained cell state.
- C. Deep Last Short Term Memory network: The cell state combines the previous state with the input-gated candidate, while the output gate controls the hidden state exposed by each layer.
- C. Deep Last Short Term Memory network: Stacked LSTMs pass the output sequence of layer l to layer l +1, while higher layers also receive the lower layer’s hidden state at time t.
- C. Deep Last Short Term Memory network: At each time t, the final-layer hidden vector is transformed into an output yt representing a probability distribution over K gesture classes.
- C. Deep Last Short Term Memory network: The final class prediction is the gesture class with the highest normalized probability after all time-step outputs are collected.
D. Network Training
The training objective minimizes a maximum-likelihood loss over labeled gesture sequences, using cross-entropy and optimization through back-propagation through time.
- The network minimizes a maximum-likelihood loss over M training gesture sequences.Each sequence is paired with its ground-truth label in the objective.
- The loss is formulated as cross-entropy using the Kronecker delta to select the ground-truth class probability.
- Back-propagation through time differentiates the objective with respect to all weights for stochastic-gradient-descent minimization.
IV. EXPERIMENTAL RESULTS
The experiments evaluate the proposed approach using a Leap Motion Controller and a specified computing setup, with joint angles assessed as recognition features.
- Experiments used a Leap Motion Controller on an Intel i5 3.2GHz system with 16GB RAM and a GeForce GTX 1050ti.
- The DLSTM network and BPTT-based stochastic-gradient-descent minimization were implemented using Keras.
- The experiments assessed joint angles as salient features for hand-gesture recognition.
A. Dataset
The authors created a 30-gesture ASL dataset because no public dataset combined many classes with hand-joint information. It contains static and dynamic gestures collected from 20 people.
- The dataset contains 30 ASL gestures, including 12 dynamic and 18 static gestures.
- The dataset was created because no public ASL dataset offered both a large number of classes and hand-joint information.
- The 1200 gesture sequences were collected from 20 people, comprising 15 males and 5 females aged 20 to 28 years.
- The gestures were selected to represent variation in joint angles and finger positions during hand movements.
B. Selection of the Optimal Number of Stacked LSTM
The experiments identify four stacked LSTM levels as the preferred architecture: it gives the best accuracy at 800 epochs and balances training time with accuracy, while deeper networks need more epochs without substantial additional benefit.
- The proposed solution is evaluated with accuracy, precision, recall, and F1-score metrics, alongside a confusion matrix.
- Four stacked LSTM levels give the best accuracy results when each architecture is trained for 800 epochs.
- 1600 epochs are used for the 5-LSTM architecture and 1800 for the 6-LSTM architecture, improving their results relative to shorter training.
- Increasing depth beyond four LSTM levels provides no substantial benefit because accuracy begins converging to a precise value.
- Four levels are selected as a compromise between training time and system accuracy.
C. Feature Effectiveness Analysis
The analysis shows that combining joint-angle features with complementary fingertip and hand-movement features improves ASL gesture classification, while similar gestures remain a key challenge. Training and test behavior is also examined through accuracy and loss curves, with test accuracy converging after extended training.
- Feature combinations: Combining ω_i and β_i features distinguishes many ASL gestures and performs better than using either feature type separately.The γ_j feature further improves classification when combined with ω_i and β_i.
- Feature combinations: Hand-movement features (u_w, v_w, z_w) cannot classify gestures alone but support high performance when combined with angle features.
- Evaluation: The section reports accuracy, precision, recall, and f1-score as evaluation metrics and presents the results in Table II.
- Ambiguous gestures: 91.5178% accuracy was achieved without grouping the visually similar 2, 6, V, and W classes.The Leap Motion Controller fails to capture their minimal joint-angle variations, motivating the grouped evaluation.
- Training behavior: After approximately 125000 iterations, the test accuracy curve converges while the plots also track maximum-likelihood loss.Accuracy represents correctly recognized training or validation instances, whereas loss sums errors across instances.
E. Comparisons
The proposed method is compared with established approaches on the SHREC dataset to assess performance across gesture-recognition settings. These experiments also indicate applicability beyond language gestures, while the conclusion frames the approach as a DLSTM system using joint-angle and fingertip-position features.
- SHREC comparison: The proposed method is compared with key state-of-the-art approaches on the SHREC dataset using its multiple acquisition-sensor settings.
- SHREC comparison: SHREC experiments indicate that the method can also handle semaphoric gestures, which define symbols for communicating with machines.
- Conclusion: The paper presents a DLSTM approach using joint angles and fingertip positions, evaluated with a large ASL gesture dataset.The conclusion also proposes expanding the public ASL dataset and integrating RGB information for ambiguous cases.