Source-linked AI summary
DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation
Biyi Fang, Jillian Co, Mi Zhang
TL;DR
Deaf people face a communication barrier with hearing people, while translating raw sign data into words and sentences remains challenging. DeepASL uses infrared sensing and deep learning for non-intrusive word- and sentence-level ASL translation, achieving 94.5% average word accuracy and 8.2% average word error rate on unseen sentences.
Problem
Deaf people face a communication barrier because few hearing people understand ASL, while existing communication alternatives have limits in cost, availability, or convenience.
Source: §1 · p. 1Method
DeepASL uses infrared sensing of skeleton joints, an HB-RNN for word-level translation, and CTC-based sequence modeling for sentence-level translation.
Source: §1 · p. 2, §5.3 · p. 7Results
94.5% average translation accuracy over 56 commonly used ASL words; 8.2% average word error rate on unseen ASL sentences and 16.1% on sentences performed by unseen users, over 100 commonly used ASL sentences.
Source: §10 · p. 12Takeaways & Limitations
DeepASL supports ubiquitous, non-intrusive translation at both word and sentence levels, with sentence translation that does not require users to pause between adjacent signs.
Source: §8 · p. 12The dataset covers only a small portion of the ASL vocabulary, limiting the demonstrated vocabulary scope.
Source: §8 · p. 12
Abstract
from arXiv · showhide
There is an undeniable communication barrier between deaf people and people with normal hearing ability. Although innovations in sign language translation technology aim to tear down this communication barrier, the majority of existing sign language translation systems are either intrusive or constrained by resolution or ambient lighting conditions. Moreover, these existing systems can only perform single-sign ASL translation rather than sentence-level translation, making them much less useful in daily-life communication scenarios. In this work, we fill this critical gap by presenting DeepASL, a transformative deep learning-based sign language translation technology that enables ubiquitous and non-intrusive American Sign Language (ASL) translation at both word and sentence levels. DeepASL uses infrared light as its sensing mechanism to non-intrusively capture the ASL signs. It incorporates a novel hierarchical bidirectional deep recurrent neural network (HB-RNN) and a probabilistic framework based on Connectionist Temporal Classification (CTC) for word-level and sentence-level ASL translation respectively. To evaluate its performance, we have collected 7,306 samples from 11 participants, covering 56 commonly used ASL words and 100 ASL sentences. DeepASL achieves an average 94.5% word-level translation accuracy and an average 8.2% word error rate on translating unseen ASL sentences. Given its promising performance, we believe DeepASL represents a significant step towards breaking the communication barrier between deaf people and hearing majority, and thus has the significant potential to fundamentally change deaf people's lives.
1 INTRODUCTION
DeepASL targets the communication barrier by offering non-intrusive ASL translation at both word and sentence levels. Its reported results cover translation accuracy, real-world conditions, and runtime across three platforms.
1 INTRODUCTION
- Existing systems may be intrusive, limited by resolution or ambient conditions, and restricted to translating one sign at a time.The paper notes that pausing between adjacent signs slows face-to-face conversations; interpreters, paper, and phone typing also have cost, availability, or convenience limitations.Source: §1 · p. 1
- DeepASL is designed for embedding in devices from wearables and phones to computers and cloud servers, with prototype applications demonstrating practical value.An envisioned wearable scenario pairs sign-to-spoken-English translation with speech recognition that projects English text through AR glasses.Source: §1 · pp. 1–2
- Infrared sensing captures signs non-intrusively, while domain knowledge, HB-RNN, and CTC support word-level modeling and end-to-end sentence translation.The CTC framework removes the need to pre-segment sentences and allows translation of sentences absent from training.Source: §1 · p. 2
- 94.5% average word-level translation accuracy; sentence translation reached 8.2% average word error rate on unseen ASL sentences and 16.1% on sentences performed by unseen users.The evaluation used 7,306 samples from 11 participants, covering 56 commonly used words and 100 sentences.Source: §1 · p. 2
- More than 91.8% word-level translation accuracy was achieved across varied lighting, body postures, and interference sources.These tests assess robustness in real-world daily communication scenarios.Source: §1 · p. 2
- 282 ms was the worst-case runtime across three platforms for both word-level and sentence translation.DeepASL also demonstrated enough inferences for daily usage on mobile and tablet platforms.Source: §1 · p. 2
2 BACKGROUND, STATE-OF-THE-ART, AND DESIGN CHOICE
ASL signs combine hand shape, movement, and the relative location of two hands, with subtle differences and real-world conditions making translation challenging. DeepASL uses Leap Motion to capture these cues contactlessly, including in poor lighting, while preserving privacy.
2.1 Characteristics of ASL
- Hand shape, hand movement, and the relative location of two hands jointly encode each ASL sign’s meaning.Subtle differences in these characteristics, along with varied lighting, posture, and environments, make translation challenging.Source: §2.1 · p. 3
2.2 State-of-the-Art ASL Translation Systems
- Wearable sensors are intrusive, RF sensing has limited hand resolution, RGB cameras struggle in poor lighting or uncontrolled backgrounds and raise privacy concerns, and Kinect lacks hand-shape resolution.These limitations constrain the practical use of existing sensing approaches for daily-life ASL translation.Source: §2.2 · p. 3
2.3 Design Choice
- Leap Motion uses infrared sensing to capture finger, palm, and forearm skeleton joints contactlessly, including in poor lighting, while preserving privacy and capturing hand shape, movement, and location.The paper compares Leap Motion with other sensing modalities for ASL translation.Source: §2.3 · p. 3, Table 1 · p. 3
3 CHALLENGES AND OUR SOLUTIONS
DeepASL addresses challenges in translating raw skeleton-joint data into words and continuous sentences. It extracts sign characteristics, organizes them with hierarchical and bidirectional RNNs, and uses CTC to translate whole sentences without pre-segmenting signs.
ASL Characteristics Extraction
- ASL’s hand shape, movement, and relative hand location remain buried in raw skeleton-joint data, so DeepASL extracts their spatio-temporal trajectories using ASL domain knowledge.The extracted characteristics provide the basis for translation models.Source: §3 · p. 4
ASL Characteristics Organization
- A hierarchical deep RNN integrates isolated low-level ASL characteristics into an organized high-level representation for translation as the sign vocabulary scales.The extracted characteristics cannot be used directly while isolated and unorganized.Source: §3 · p. 4
Similarity between Different Signs
- Because signs such as “want” and “what” can share similar trajectory beginnings, a bidirectional RNN uses past and future information at each point to infer the sign.This contrasts with traditional unidirectional RNNs, which use only past information at each time point.Source: §3 · p. 4, Figure 4 · p. 4
ASL Sentence Translation
- CTC computes the probability of a whole ASL sentence directly, removing the need to pre-segment signs and pause between adjacent signs.Existing sentence-translation frameworks that require pre-segmenting individual words restrict translation to one sign at a time.Source: §3 · p. 4
- DeepASL is presented as the first ASL translation framework to address these challenges while achieving accurate word- and sentence-level translation.The stated challenges include extracting and organizing sign characteristics, distinguishing similar signs, and translating sentences without pre-segmentation.Source: §3 · p. 4
4 SYSTEM OVERVIEW
DeepASL processes skeleton-joint sequences through layers that extract sign characteristics, translate words with an HB-RNN, and translate full sentences end-to-end with CTC.
4 SYSTEM OVERVIEW
- Leap Motion captures temporal sequences of 3D coordinates for finger, palm, and forearm skeleton joints during signing.These sequences form the system’s input.Source: §4 · p. 4
- DeepASL extracts hand shape, movement, and relative hand location from each frame, producing spatio-temporal trajectories that an HB-RNN models for word-level translation.The HB-RNN models both the spatial structure and temporal dynamics of the trajectories.Source: §4 · p. 4
- A top-layer CTC framework uses probabilistic dependencies between words to translate a complete sentence end-to-end without requiring pauses between adjacent signs.The framework avoids requiring pre-segmentation of the sentence into signs.Source: §4 · p. 4
5 SYSTEM DETAILS
DeepASL transforms Leap Motion skeleton-joint data into ASL characteristics, then uses hierarchical bidirectional recurrent modeling for word translation and CTC for end-to-end sentence translation.
5.1 ASL Characteristics Extraction
- After smoothing noisy skeleton-joint data, DeepASL extracts hand shape, hand movement, and relative hand location as four spatio-temporal trajectories.The four trajectories represent right- and left-hand shape and movement; the left-hand shape trajectory also encodes the left hand's location relative to the right.Source: §5.1 · p. 5
5.2 Word-Level ASL Translation
- HB-RNN hierarchically integrates low-level ASL characteristics into a high-level representation for single-sign word translation.Bidirectional recurrent layers process hand-shape and hand-movement trajectories, which are fused first by hand and then across both hands.Source: §5.2.2 · p. 6, Figure 8 · p. 6
5.3 Sentence-Level ASL Translation
- CTC enables end-to-end sentence translation without word pre-segmentation or post-processing, addressing the need to pause between adjacent signs.The model marginalizes over label sequences equivalent to the target and removes adjacent duplicate labels and blanks from the inferred sequence.Source: §5.3 · p. 7
6 EVALUATION
DeepASL was evaluated on word and sentence translation, robustness in field conditions, and system performance across hardware platforms. It achieved 94.5% average word accuracy and 8.2% word error rate on unseen sentences.
6.1 Experimental Setup
- The evaluation used 56 selected common ASL words, 100 meaningful sentences, and three datasets; leave-one-subject-out testing assessed generalization across participants.The sentence set was built from a four-word template, with 100 meaningful sentences selected from 256 possible combinations; unseen sentences were tested with ten-fold cross-validation.Source: §6.1.1 · p. 7, §6.1.3 · p. 8, §6.1.4 · p. 8
6.2 Word-Level Translation Performance
- 94.5% average word accuracy and 8.2% average word error rate on unseen sentences; unseen-user sentences had a 16.1% average word error rate.Word accuracy was measured across 11 participants; unseen-sentence evaluation used sentences excluded from training.Source: §6.2 · p. 8, Figure 11 · p. 9, Figure 13 · p. 9
6.3 The Necessity of Model Components
- HB-RNN outperformed four comparative models across Top-K metrics, with Top-1 accuracy gains of 5.1%, 5.0%, 3.4%, and 0.8%.The comparison matched models to roughly the same number of parameters.Source: Table 5 · p. 9, §6.3 · p. 9
6.4 Sentence-Level Translation Performance
- 16.1 ± 3.7% Top-1 word error rate on unseen participants; evaluation used leave-one-subject-out validation without restricting sentence length or word order or using a language model.The authors describe 16 candidate ASL words per frame and an average of 0.64 edits for a four-word sentence.Source: §6.4.1 · p. 9
6.5 Robustness of ASL Translation in the Field
- Field accuracy was 96.8 ± 3.1% in poor lighting, 91.8 ± 4.1% in sunlight, 92.2 ± 3.0% while walking, and 94.9 ± 4.5% while standing.People-in-the-scene and multi-device interference yielded 94.7 ± 3.0% and 94.1 ± 1.3% accuracy, respectively.Source: §6.5 · pp. 9–10, Table 6 · p. 10
6.6 System Performance
- Inference took less than 282 ms on all three CPUs; CPU inference also supported 29× to 57× more estimated TX1 inferences than GPU inference, depending on model.The authors report higher memory use for sentence inference than for two-hand word inference, and higher use for two-hand than one-hand word inference. Hololens runtime was not tested because it could not host USB clients.Source: §6.6.1 · p. 10, §6.6.2 · p. 10, Figure 15 · p. 10
7 APPLICATIONS
The paper presents two prototype applications for DeepASL: an AR-based Personal Interpreter and a tablet-based ASL Dictionary.
7 APPLICATIONS
- The prototypes demonstrate application scenarios for DeepASL, including communication through AR and sign-based word lookup.The applications are presented as demonstrations of the technology’s practical value.Source: §7 · p. 11, §7.1 · p. 11, §7.2 · p. 11
7.1 Application#1: Personal Interpreter
- The Personal Interpreter supports two-way communication: it converts speech to text for AR display and translates the deaf user’s ASL into spoken language.The prototype used a Hololens; ASL recognition results were sent to it over TCP/IP.Source: §7.1 · p. 11, Figure 16 · p. 11
7.2 Application#2: ASL Dictionary
- The ASL Dictionary accepts a performed sign and displays its meaning in real time, addressing the difficulty of indexing ASL gestures for lookup.The prototype was implemented as a Microsoft Surface tablet application.Source: §7.2 · p. 11, Figure 17 · p. 12
8 DISCUSSION
DeepASL combines infrared sensing with sign-level and sentence-level translation capabilities. The authors also identify limited vocabulary coverage in their dataset and describe a crowdsourcing initiative.
Impact on ASL Translation Technology
- DeepASL combines infrared and skeleton-joint sensing with HB-RNN sign modeling and CTC sentence translation, without requiring pauses between adjacent signs.The authors characterize it as the first technology enabling ubiquitous, non-intrusive ASL translation at both word and sentence levels.Source: §8 · p. 12
Initiative on ASL Sign Data Crowdsourcing
- The dataset covers only a small portion of ASL vocabulary despite months of collection, so the authors made their data-collection toolkit publicly available.They hope crowdsourcing will expand coverage to more words and everyday sentences.Source: §8 · p. 12
9 RELATED WORK
The related work covers sign-language translation systems using several sensing modalities and broader mobile-sensing research. DeepASL is positioned as an infrared-sensing application with word- and sentence-level translation.
Sign Language Translation Systems
- Prior sign-language translation systems used wearable motion/EMG/bend sensors, RGB cameras, and Kinect; examples include a wrist-worn device recognizing 40 signs and a hat-mounted RGB camera/HMM recognizing 40 ASL words.Source: §9 · p. 12
- Earlier Leap Motion systems cited by the authors recognize static ASL signs from hand shape, whereas DeepASL captures hand shape and movement for dynamic signs and enables end-to-end sentence translation.The cited prior systems are.Source: §9 · p. 12
Mobile Sensing Systems
- The mobile-sensing work spans accelerometers, microphones, physiological sensors, and wireless radio; this paper explores infrared sensing for ASL translation.The authors describe infrared as a non-intrusive, high-resolution sensing scheme complementary to existing mobile-sensing systems.Source: §9 · p. 12
10 CONCLUSION
DeepASL provides ubiquitous, non-intrusive ASL translation at word and sentence levels. It reports 94.5% average word-level accuracy and 8.2% and 16.1% average word error rates for unseen sentences and unseen users, respectively.
10 CONCLUSION
- 94.5% average translation accuracy for 56 commonly used ASL words; sentence-level word error rates are 8.2% for unseen sentences and 16.1% for sentences performed by unseen users.Source: §10 · p. 12
- DeepASL enables ubiquitous and non-intrusive ASL translation at both word and sentence levels.Source: §10 · p. 12
- The sentence-level results cover 100 commonly used ASL sentences, while the word-level result covers 56 commonly used ASL words.Source: §10 · p. 12