Source-linked AI summary
Real-time Sign Language Fingerspelling Recognition using Convolutional Neural Networks from Depth map
Byeongkeun Kang, Subarna Tripathi, Truong Q. Nguyen
TL;DR
Static ASL fingerspelling is challenging because signs can look alike and vary across signers and viewpoints. This work classifies 31 alphabet and number signs from depth maps with CNNs, reporting 99.99% accuracy for observed signers and 83.58%–85.49% for new signers.
Problem
Static ASL fingerspelling remains challenging because visually similar signs vary across signers and camera viewpoints.
Source: §1 · p. 1Method
Results
Accuracy was 99.99% when training and validation included test-subject samples, and fine-tuning reached 83.58% and 85.49% for new signers under two data splits.
Source: §4 · pp. 3–4Takeaways & Limitations
The authors conclude that color-image classification pretraining helps on depth images, and they make the trained model and dataset available.
Source: §5 · p. 5Hand segmentation assumes the user's hand is the closest object to the camera.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Sign language recognition is important for natural and convenient communication between deaf community and hearing majority. We take the highly efficient initial step of automatic fingerspelling recognition system using convolutional neural networks (CNNs) from depth maps. In this work, we consider relatively larger number of classes compared with the previous literature. We train CNNs for the classification of 31 alphabets and numbers using a subset of collected depth data from multiple subjects. While using different learning configurations, such as hyper-parameter selection with and without validation, we achieve 99.99% accuracy for observed signers and 83.58% to 85.49% accuracy for new signers. The result shows that accuracy improves as we include more data from different subjects during training. The processing time is 3 ms for the prediction of a single image. To the best of our knowledge, the system achieves the highest accuracy and speed. The trained model and dataset is available on our repository.
1. Introduction
The paper targets static ASL fingerspelling, where visually similar signs and differences among signers and viewpoints make recognition challenging. It uses depth sensing and CNNs as a route to real-time, accurate recognition.
1. Introduction
- Sign language recognition is challenging because it involves hand pose, movement, facial expression, and body posture across many signs.The paper notes that signs may have very similar hand poses.Source: §1 · p. 1
- Static ASL signs can look alike, and their appearance varies across signers and camera viewpoints.Some signs differ only by thumb position.Source: §1 · p. 1
- Depth sensing and CNNs are used to pursue real-time, accurate sign language recognition.The introduction identifies depth sensors as a source of additional information for improving accuracy and/or processing time.Source: §1 · p. 1
2. Related Work
Prior systems addressed gesture, other sign-language, and ASL fingerspelling tasks with varied methods and scopes. This paper distinguishes its work through 31 sign classes, CNN fine-tuning, and a publicly available dataset.
2. Related Work
- Related systems include small gesture-class recognizers, Italian and British sign-language systems, and recognition of short ASL sentences.The cited gesture systems considered six classes; the British system recognized fingerspelled words from video, while the ASL sentence system handled three-to-five-word sentences.Source: §2 · pp. 1–2
- Earlier ASL fingerspelling systems used wavelet features, Gabor filters, random forests, or hand-joint features, with different signer and class scopes.Pugeault et al. reported 69% using depth; Dong et al. reported 70% for new signers.Source: §2 · p. 2
- The paper presents 31 alphabet and number classes, compared with the 24 classes reported for the cited state-of-the-art approach.The figure shows ASL fingerspelling alphabets and numbers, and the dataset follows formal sign demonstrations [15].Source: §2 · p. 2, Figure 1 · p. 2
- The work fine-tunes CNN parameters pretrained on 1.28 million color images and contributes a publicly available dataset.The paper notes that existing sign-language datasets are limited in quantity and quality.Source: §2 · p. 2, Figure 1 · p. 2
3. Method
The method collects depth images across subjects and viewpoints, segments hands using depth structure, and classifies processed images with a CNN under several training and data-split settings.
3.1. Dataset
- 31,000 depth maps cover 31 signs across five subjects, with 1,000 images per sign and varied hand viewpoints.J and Z are omitted because they require temporal information; (2/V) and (6/W) each share one class because they are distinguished by context.Source: §3.1 · p. 2, Figure 4 · p. 3
3.2. Hand Segmentation
- Assuming the hand is nearest the camera, segmentation uses the closest-depth region and surrounding depth voids to extract connected components.A black wristband creates depth voids; the resulting hand bounding box is scaled to 227 × 227, then padded to the 256 × 256 input size.Source: §3.2 · pp. 2–3, Figure 2 · p. 3
3.3. Classification
- The classifier uses CaffeNet, a CNN architecture nearly equivalent to AlexNet, with five convolutional and three fully connected layers.Three max-pooling layers are also used, with rectified linear units after layers except the last.Source: §3.3 · p. 3
- Each preprocessed depth image is mean-subtracted and forward-propagated to extract a 4096-dimensional final-layer feature vector.Source: §3.3 · p. 3
- Five operating modes vary retraining versus fine-tuning and whether subject data are separated across training, validation, and testing.Fine-tuning starts from ILSVRC2012-pretrained CNN weights and replaces the final layer with 31 output classes.Source: §3.3 · p. 3
4. Experimental Results
The experiments report strong results for both observed and new signers, with fine-tuning outperforming re-training in the held-out-signer settings. The system also reports faster prediction and higher accuracy than prior work despite covering more classes.
4. Experimental Results
- The system achieves 99.99% accuracy when training and validation data include samples corresponding to the test subject.Source: §4 · p. 3
- About 10–15% improvement over the previous state-of-the-art is reported despite more classes and subjects, with 3 ms per-image prediction.The comparison covers gesture and ASL fingerspelling systems.Source: Table 1 · p. 4, §4 · p. 5
- E, M, N, and T have low accuracy for both this method and prior systems because they differ only by thumb position.The table gives detailed accuracy results for each alphabet or number.Source: Table 2 · p. 4, §4 · p. 5
- Using 25% of the dataset for training, validation, and testing, re-training averaged 75.18% (3/1/1 subjects) and 78.39% (4/1); fine-tuning averaged 83.58% and 85.49%, respectively.Source: §4 · p. 4
- Across all possible training, validation, and test subject combinations, average accuracy improves about 2∼3% as the number of training subjects increases, suggesting likely gains for new-subject performance.Training-phase iterations are fixed at 8000 for re-training and 4000 for fine-tuning.Source: §4 · p. 4
- The system uses depth alone to avoid illumination and skin-pigment differences and a calibration process, although prior work reported improved recognition from combining color and depth.Source: §4 · p. 5
5. Conclusion
The paper presents a depth-image CNN system for recognizing 31 ASL alphabet and number signs, reports real-time performance and state-of-the-art accuracy across signers, and shares its dataset and trained model.
5. Conclusion
- The approach classifies 31 alphabet and number signs from depth images using CNNs, with real-time performance and state-of-the-art accuracy across different signers.Source: §5 · p. 5
- The authors collect and share a depth-image dataset for ASL fingerspelling recognition.Source: §5 · p. 5
- The authors conclude that pre-training on color-image classification is helpful for the different input type of depth images.Source: §5 · p. 5