Source-linked AI summary
Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison
Dongxu Li, Cristian Rodriguez Opazo, Xin Yu, Hongdong Li
TL;DR
Word-level ASL recognition needs larger and more varied datasets than prior small-scale benchmarks provide. The paper introduces WLASL and compares appearance- and pose-based deep models, including a temporal graph network. On 2,000 words, the approaches achieve comparable performance, reaching up to 62.63% classification accuracy, while sample scarcity remains a training limitation.
Problem
Prior sign-language datasets and methods are often small-scale, limiting evaluation of recognition across the thousands of vocabulary classes and signer variations needed for practical ASL recognition.
Method
The paper constructs WLASL from RGB videos and benchmarks holistic appearance and 2D pose models, proposing a temporal graph convolutional network for spatial-temporal pose dependencies.
Results
62.63% classification performance was reached on 2,000 words, with pose-based and appearance-based approaches achieving comparable performance.
Takeaways & Limitations
WLASL provides a large-scale ASL benchmark and baseline methods for evaluating word-level sign recognition algorithms.
Takeaways & Limitations
Training remains difficult because the average number of training examples per gloss is relatively small compared with large-scale generic activity-recognition datasets.
Abstract
from arXiv · showhide
Vision-based sign language recognition aims at helping deaf people to communicate with others. However, most existing sign language datasets are limited to a small number of words. Due to the limited vocabulary size, models learned from those datasets cannot be applied in practice. In this paper, we introduce a new large-scale Word-Level American Sign Language (WLASL) video dataset, containing more than 2000 words performed by over 100 signers. This dataset will be made publicly available to the research community. To our knowledge, it is by far the largest public ASL dataset to facilitate word-level sign recognition research. Based on this new large-scale dataset, we are able to experiment with several deep learning methods for word-level sign recognition and evaluate their performances in large scale scenarios. Specifically we implement and compare two different models,i.e., (i) holistic visual appearance-based approach, and (ii) 2D human pose based approach. Both models are valuable baselines that will benefit the community for method benchmarking. Moreover, we also propose a novel pose-based temporal graph convolution networks (Pose-TGCN) that models spatial and temporal dependencies in human pose trajectories simultaneously, which has further boosted the performance of the pose-based method. Our results show that pose-based and appearance-based models achieve comparable performances up to 66% at top-10 accuracy on 2,000 words/glosses, demonstrating the validity and challenges of our dataset. Our dataset and baseline deep models are available at \url{https://dxli94.github.io/WLASL/}.
1. Introduction
Word-level ASL recognition is difficult because subtle motion and pose differences can change meaning, while practical use requires thousands of vocabulary classes and varied examples. WLASL addresses these gaps with a large-scale RGB video dataset and compares appearance- and pose-based methods, including a temporal graph model.
- Word-level recognition is challenging because sign meaning depends on body motions, manual movements, and head poses.
- Thousands of daily-use sign vocabulary classes create a scalability challenge beyond typical gesture and action recognition datasets.
- Existing small-scale datasets inadequately capture lexical ambiguity and signer variation, motivating sufficient classes and training examples.
- WLASL compares holistic appearance and 2D pose approaches and introduces a temporal graph convolutional model for joint spatial-temporal pose dependencies.
- 62.63% classification performance was reached on 2,000 words, with pose- and appearance-based approaches performing comparably.
2. Related Work
Prior word-level sign-language datasets and recognition methods are fragmented across languages, vocabularies, and scales. WLASL provides a large-scale ASL benchmark with RGB videos and baseline methods intended to make evaluations more comparable.
- 2.1. Sign Language Datasets: Existing public word-level ASL datasets vary substantially in vocabulary, samples, signers, and whether signs correspond directly to English words.
- 2.1. Sign Language Datasets: Some datasets have broad vocabulary but too few examples per gloss, limiting their suitability for thousand-way classifier training.
- 2.1. Sign Language Datasets: Previously mentioned datasets fail to capture word-level recognition difficulties because they contain insufficient instances and signers.
- 2.2. Sign Language Recognition Approaches: Existing recognition models are often trained on private or small-scale datasets with fewer than one hundred words.
- 2.2. Sign Language Recognition Approaches: Prior methods combine feature extraction, temporal-dependency modeling, and classification using handcrafted features, HMMs, DTW, CNNs, GRUs, or 3D CNNs.
- 2.2. Sign Language Recognition Approaches: WLASL contributes a large-scale RGB-only ASL dataset and state-of-the-art baselines for evaluating future recognition methods.
3. Our Proposed WLASL Dataset
WLASL is constructed from Internet-based ASL resources with filtering, annotation, and signer-aware metadata, then organized into vocabulary-scale subsets. Its largest subset contains substantial video coverage but still faces sample scarcity as vocabulary grows.
- WLASL combines automatic detection and manual annotation to provide glosses, temporal boundaries, body boxes, signer information, and dialect or variation labels.
- The dataset draws on educational sign-language websites and YouTube tutorials to construct a large-scale signer-independent ASL collection.
- Videos with multiword glosses or fewer than seven samples are removed to retain word-level labels and enough data for splits.
- Video durations range from 0.36 to 8.12 seconds, averaging 2.41 seconds across the dataset.
- WLASL is organized into WLASL100, WLASL300, WLASL1000, and WLASL2000 using the top 100, 300, 1,000, and 2,000 glosses.
- 21,083 video samples spanning around 14 hours form WLASL2000, averaging 10.5 samples per gloss.
4. Method Comparison on WLASL
The paper compares appearance-based and pose-based deep models for sign recognition, examining how different architectures represent visual and human-pose information. The methods are used to assess WLASL and sign-recognition performance at scale.
- The method comparison evaluates image-appearance and pose-based baseline models for word-level sign recognition.
4.1. Image-appearance based Baselines
The paper compares image-appearance baselines that model spatial and temporal information in word-level sign videos. These include a 2D Conv RNN using VGG16 and GRU, and a fine-tuned I3D network.
- Word-level sign recognition must capture detailed hand orientations and arm movements rather than rely on background context.
- The 2D Conv RNN extracts spatial features with ImageNet-pretrained VGG16 and models temporal dependencies with a stacked GRU.
- The 2D Conv RNN trains on up to 50 consecutive frames, applies cross-entropy at every time step and after average pooling, and uses all frames at testing.
- I3D provides a second appearance-based baseline by using 3D convolutions to represent frames and their temporal relationships hierarchically.
- I3D is fine-tuned to capture sign-specific hand shapes, orientations, and arm movements, with its final classification layer adjusted to each WLASL subset.
4.2. Pose-based Baselines
The pose-based baselines represent signing through human keypoints and temporal modeling. The proposed Pose-TGCN additionally learns spatial and temporal dependencies among pose trajectories with graph convolutions.
- Pose-based sign recognition requires modeling spatial and temporal relationships among all keypoints, beyond using joint locations alone.
- The first pose baseline uses an RNN to model temporal pose sequences and feeds its representation to sign recognition.
- The pose input contains 55 body and hand 2D keypoints extracted with OpenPose, concatenated as coordinates and processed by a two-layer stacked GRU.
- Pose-TGCN models spatial and temporal dependencies simultaneously and represents motion through holistic trajectories of body keypoints.
- The graph treats the body as fully connected, uses a weighted adjacency matrix, and applies trainable graph convolutions to keypoint features.
- A residual graph convolutional block stacks two graph convolutional layers, while Pose-TGCN averages temporal features before softmax classification.
4.3. Training and Testing Protocol
The experiments use standardized preprocessing, optimization, data splitting, and top-K evaluation protocols. The evaluation emphasizes multiple predicted labels because similar gestures can produce classification errors.
- Training uses resized frames, 224×224 random crops, and video-consistent horizontal flipping for the appearance models.
- VGG-GRU, Pose-GRU, and Pose-TGCN use randomly selected consecutive frame sequences during training, while testing uses full videos.
- All models are implemented in PyTorch, trained for 200 epochs with Adam, and I3D uses Adam because stochastic gradient descent did not converge during fine-tuning.
- Gloss samples are split into training, validation, and testing sets at a 4:1:1 ratio, with every split containing at least one sample per gloss.
- Evaluation reports mean top-K classification accuracy for K = {1, 5, 10} over all sign instances.
- Top-K predictions are considered appropriate because similar gestures can cause errors that contextual information may rectify.
4.4. Discussion
On WLASL, pose-based and appearance-based baselines perform similarly on the largest vocabulary, while Pose-TGCN improves pose recognition by modeling spatial and temporal keypoint relationships. Performance is higher on smaller vocabularies, revealing challenges from vocabulary size, sign ambiguity, and limited samples per class.
- Baseline performance: Pose-TGCN improves classification accuracy over Pose-GRU by capturing both spatial and temporal relationships among body keypoints.Pose-GRU mainly models temporal dependencies, whereas Pose-TGCN models spatial and temporal dependencies simultaneously.
- Baseline performance: I3D outperforms VGG-GRU among appearance-based models, attributed to larger network capacity and pretraining on ImageNet and Kinetics.
- Baseline performance: Pose-TGCN achieves comparable top-5 and top-10 accuracy to the larger I3D model on WLASL2000.The comparison supports the effectiveness of Pose-TGCN for encoding human motion information.
- Effect of vocabulary size: Increasing the number of classes lowers performance because models face harder tasks and more ambiguous signs.Tests across WLASL subsets show higher accuracy when the test class number is smaller.
- Effect of sample numbers: As vocabulary size increases, limited training examples per gloss make network training more difficult, despite WLASL providing more samples per gloss than other sign datasets.WLASL100 averages twice as many training samples per gloss as WLASL2000.
5. Conclusion
The paper introduces WLASL, a large-scale word-level ASL dataset, and evaluates deep learning methods on it. Comparisons indicate that large-vocabulary recognition requires more advanced methods, while the annotations may support future sentence- and story-level translation.
- WLASL covers a wide range of daily ASL words and is used to evaluate deep learning methods for word-level recognition.
- The authors identify few-shot learning as a promising direction for recognition on large-scale datasets.They motivate this direction by the difficulty and expense of labeling many samples per class with specialized domain knowledge.
- Word-level annotations are proposed as a basis for future sentence-level and story-level machine sign translation.