Source-linked AI summary
Using Deep Convolutional Networks for Gesture Recognition in American Sign Language
Vivek Bheda, Nelu Radpour
TL;DR
ASL sign recognition remains understudied, and fingerspelling makes individual-letter recognition important for spelling words without signs. This paper trains a CNN from scratch to classify ASL letters and digits from images. It reports 82.5% alphabet accuracy and 97% validation accuracy on digits with the NZ ASL dataset, while noting data and representation constraints on the approach.
Problem
ASL recognition is an understudied area, and fingerspelling makes individual-letter recognition important for spelling words without signs.
Source: §1 · p. 1Method
The paper trains a CNN from scratch to classify ASL alphabet letters and digits from images.
Source: §3.1 · p. 2Results
On the NZ ASL dataset, the model achieved 82.5% accuracy on alphabet gestures and 97% validation set accuracy on digits.
Source: §5 · p. 3Takeaways & Limitations
The authors say their simple-camera approach has potential if enough substantial training data is provided and can be continuously added through the processing pipeline.
Source: §6 · p. 4Results on self-generated data were hindered by skin-color and lighting variation, and the simple-camera approach is conditional on enough substantial training data.
Source: §6 · p. 4
Abstract
from arXiv · showhide
In the realm of multimodal communication, sign language is, and continues to be, one of the most understudied areas. In line with recent advances in the field of deep learning, there are far reaching implications and applications that neural networks can have for sign language interpretation. In this paper, we present a method for using deep convolutional networks to classify images of both the the letters and digits in American Sign Language.
1. Introduction
The introduction frames ASL recognition as an understudied communication task, with letter recognition especially relevant because fingerspelling is used for names and other words without signs. The paper situates its alphabet and digit figures in a system where 22 handshapes correspond to 26 letters and the 10 digits can be signed on one hand.
1. Introduction
- ASL has 22 handshapes corresponding to 26 alphabet letters, while the 10 digits can be signed on one hand.Figures 1 and 2 present the alphabet and numbers, respectively.Source: §1 · p. 1, Figure 1 · p. 1, Figure 2 · p. 1
- Fingerspelling supports spelling words without signs, making recognition of individual letters important for interpreting names, places, titles, brands, and other items.The examples are explicitly non-exhaustive.Source: §1 · p. 1
2. Related Work
Related work describes sign-recognition systems using depth and contour information, custom color gloves, and CNNs applied to ASL letter images. The paper positions its work alongside prior image-based CNN classification and depth-enabled approaches.
2. Related Work
- Depth-sensing cameras can provide depth and motion profiles for gestures, making sign recognition easier.The related work also notes that depth-sensing technology is becoming more widely available.Source: §2 · p. 1
- Custom-designed color gloves have been used to make feature extraction more efficient by helping identify and classify gestural units.Source: §2 · p. 1
- Earlier basic-camera datasets contained images without depth or contour information, and CNN classification of ASL letter images had been attempted with pre-trained GoogLeNet.Source: §2 · p. 2
3. Method
The method uses supervised learning with mini-batch stochastic gradient descent to classify ASL letters and digits from resized images. Its CNN is trained from scratch and combines convolutional, pooling, dropout, and fully connected layers.
3. Method
- The task is classifying every ASL letter and digits 0–9 from fixed-size images resized to 200 by 200 pixels using mini-batch stochastic gradient descent.Source: §3 · p. 2
3.1 Architecture
- The network was trained from scratch rather than using transfer learning, with three convolutional groups followed by two fully connected groups and a final output layer.Each convolutional group contains two convolutional layers, max-pooling, and dropout; the fully connected groups also use dropout.Source: §3.1 · p. 2, Figure 3 · p. 2
4. Data
The data section contrasts a small self-generated collection with a premade dataset, then describes preprocessing and augmentation. Cross-source validation performed poorly, while augmentation improved performance, especially on the premade dataset.
4. Data
- The self-generated dataset contained 25 images from five people per alphabet letter and digits 1–9, and a pipeline was developed to let people add images.The images were captured by the researchers and backgrounds removed using background-subtraction techniques.Source: §4 · p. 2, §4.1 · p. 2
4.1 Preprocessing
- Validation accuracy fell drastically when training and validation used images from different sources, so the premade dataset was used to train the network.The self-generated images were especially prone to differences in lighting, skin color, and capture environment because they were not collected in a controlled setting.Source: §4.1 · p. 2, §4 · p. 2
4.2 Data Augmentation
- Augmentation improved performance by nearly 20% on the premade dataset, while small image transformations increased accuracy by approximately 0.05.Transformations rotated images by 20 degrees and translated them by 20% on both axes; horizontal flipping was also used, and augmentation helped more with better, more representative initial data.Source: §4.2 · p. 3
5. Results
On the NZ ASL dataset, the reported accuracies were 82.5% for alphabet gestures and 97% validation-set accuracy for digits; results were lower on the self-generated dataset. The evaluation also compares the network with previous methods.
5. Results
- 82.5% accuracy on alphabet gestures and 97% validation set accuracy on digits were observed using the NZ ASL dataset.Source: §5 · p. 3, Figure 8 · p. 3, Figure 9 · p. 3
- 67% accuracy on alphabet letters and 70% on digits were observed on the less-uniform self-generated dataset.The authors attribute the lower accuracy largely to lighting and skin-tone variation, compared with studio-collected data.Source: §5 · p. 3, §5.1 · p. 4
- Alphabet gestures converged in approximately 25 minutes, while digits converged in nearly 10 minutes.Source: §5 · p. 3
5.1 Evaluation
- Both datasets were trained with categorical cross-entropy loss.Source: §5.1 · p. 4
- The authors say the network performed quite well against previous methods; RF-JA used color gloves and a depth-sensing Kinect, while Stanford lacked background subtraction.The authors suggest Stanford’s lack of background subtraction likely explains the higher accuracy.Source: §5.1 · p. 4, Figure 12 · p. 4
6. Conclusions and Future Work
The paper sees potential for ASL classification with simple cameras, conditional on substantial training data. Future work aims to incorporate sequential context and improved representations into a broader ASL-to-English translation system.
6. Conclusions and Future Work
- The simple-camera approach may contribute to a scalable solution if substantial training data is available; lighting and skin-color variation hindered the self-generated data.The authors note that depth-sensing cameras can address this problem but may be costly and not widely accessible.Source: §6 · p. 4
- Future implementations plan to use structured PGMs to model letter-occurrence probabilities from sequential context, extending classification toward part-of-word tagging.The paper gives letter-to-letter context as an example and cites related contextual gesture work and phonological processing.Source: §6 · p. 4
- The authors aim for a holistic translation system with features recognized from simple gesture images, exploring zero-shot and one-shot learning and possible language-model error correction.They identify representation quality as important to algorithm performance.Source: §6 · pp. 4–5