Source-linked AI summary
MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language
Hamid Reza Vaezi Joze, Oscar Koller
TL;DR
Sign language recognition lacks large, realistic, labeled datasets suitable for modern deep learning, despite its multimodal and asynchronous nature. The paper introduces MS-ASL with signer-independent evaluation and proposes I3D for recognition, reporting a large-margin advantage over prior state-of-the-art methods.
Problem
Public large-scale ASL resources suitable for evaluating modern deep learning approaches are scarce, while recognition requires substantial labeled data.
Method
The paper constructs MS-ASL from realistic public videos, provides signer-independent subsets, evaluates baseline methods, and applies I3D to sign recognition.
Results
I3D outperforms current state-of-the-art methods by a large margin on the proposed benchmark.
Takeaways & Limitations
MS-ASL provides a large, challenging benchmark for studying signer-independent ASL recognition in realistic recording conditions.
Takeaways & Limitations
The dataset includes repetitive signs, dialectal variation, signer independence, and substantial visual variability across recordings.
Abstract
from arXiv · showhide
Sign language recognition is a challenging and often underestimated problem comprising multi-modal articulators (handshape, orientation, movement, upper body and face) that integrate asynchronously on multiple streams. Learning powerful statistical models in such a scenario requires much data, particularly to apply recent advances of the field. However, labeled data is a scarce resource for sign language due to the enormous cost of transcribing these unwritten languages. We propose the first real-life large-scale sign language data set comprising over 25,000 annotated videos, which we thoroughly evaluate with state-of-the-art methods from sign and related action recognition. Unlike the current state-of-the-art, the data set allows to investigate the generalization to unseen individuals (signer-independent test) in a realistic setting with over 200 signers. Previous work mostly deals with limited vocabulary tasks, while here, we cover a large class count of 1000 signs in challenging and unconstrained real-life recording conditions. We further propose I3D, known from video classifications, as a powerful and suitable architecture for sign language recognition, outperforming the current state-of-the-art by a large margin. The data set is publicly available to the community.
1 Introduction
Sign language recognition needs substantial labeled video data for vision-based models, but public large-scale ASL resources are scarce. MS-ASL addresses this gap with a broad, signer-independent benchmark and an I3D-based approach.
- Motivation: Vision-based sign recognition avoids specialized cameras, sensors, and gloves, enabling more general usage.Deep learning has also increased the amount of training data needed for strong models.
- Research gap: Public large-scale ASL resources were insufficient for evaluating recent deep learning approaches.The paper frames this shortage as a central barrier to applying current computer vision methods.
- Contributions: MS-ASL provides over 200 signers, signer-independent sets, 1000 signs, and challenging unconstrained recording conditions.The dataset is designed to support realistic generalization studies.
- Contributions: I3D is proposed as a suitable sign-language-recognition architecture and is reported to outperform previous state-of-the-art methods by a large margin.The paper also evaluates several existing approaches as baselines.
2 Previous Works
Earlier sign-language datasets and recognition systems often relied on constrained sensing, limited vocabularies, few signers, or non-reproducible resources. The paper positions MS-ASL as a larger, publicly available benchmark for signer-independent recognition in unconstrained settings.
- Recognition methods: Many earlier recognition systems used sensor gloves, depth cameras, multiple cameras, colored gloves, or other specialized equipment.These setups constrain applicability compared with single-camera vision-based recognition.
- Sign language datasets: Existing corpora covered vocabularies ranging from 50 to 483 signs, often with few signers or controlled recording conditions.Examples include Purdue RVL-SLLL, RWTH-BOSTON, and SIGNUM resources.
- Sign language datasets: Some multilingual corpora were not well curated or consistently defined for reproducible research, limiting fair comparisons.The GSL Lemmas Corpus was reported with different vocabulary counts across studies.
- Benchmark gap: RWTH-PHOENIX-Weather 2014 and DEVISIGN were identified as publicly available datasets large enough for recent deep learning, but lacking signer variety and natural recording challenges.MS-ASL is introduced to address these limitations for ASL.
- Benchmark gap: MS-ASL contributes a publicly available ASL benchmark covering over 200 signers, signer-independent sets, and unconstrained recordings.This directly targets limitations identified in prior datasets.
Proposed ASL Data Set
MS-ASL is constructed from realistic public ASL videos and organized into signer-independent subsets spanning 100 to 1000 signs. Its natural variation and ambiguity make it a challenging benchmark, evaluated with class-balanced top-one and top-five accuracy.
- Data sources: Public ASL videos provide variation in viewpoint, background, lighting, positioning, dialect, and signer identity.These properties make the sources realistic but challenging for dataset construction.
- Annotation: Labels were obtained from OCR, video captions, descriptions, and titles, with different procedures for longer and shorter videos.The pipeline used printed labels, temporal captions, and titles as available.
- Signer processing: Bounding-box detection and face recognition tracked signers, filtered static-image descriptions, and split videos containing sequential signers.Signer tracking supported construction of signer-independent partitions.
- Dataset organization: The dataset releases ASL100, ASL200, ASL500, and ASL1000 subsets covering the 100, 200, 500, and 1000 most frequent words.Each subset has train, test, and validation sets with signer-independent splits.
- Dataset challenges: The benchmark includes repetitive performances, dialectal variants, signer independence, and visual variation in backgrounds, lighting, clothing, and camera viewpoint.These factors make it more challenging than conventional video-classification datasets.
- Evaluation: Average per class accuracy and average per class top-five accuracy address class imbalance and contextual ambiguity in ASL.Top-five accuracy counts a prediction as correct when the ground-truth label appears among the five guesses.
4 Baseline Methods
The baseline methods treat isolated sign language recognition as video classification and cover 2D recurrent, body key-point, and 3D convolutional approaches. Experiments use signer-centered video inputs and compare these methods on the proposed ASL data sets.
- Baseline Methods: Three baseline families use 2D convolution with recurrence, body key-points, or 3D convolution for isolated sign recognition.The paper frames isolated sign recognition as a video classification task and implements at least one method from each action-recognition category.
- Implementation: Inputs are signer bounding boxes resized to 224×224 pixels, with 64-frame temporal windows and consistent horizontal flipping during augmentation.Bounding boxes are extracted with SSD; shorter videos may be elongated by repeating their first or last frame.
- 2D-CNN: The 2D-CNN baseline extracts features from each video frame and uses LSTM layers to encode temporal ordering and long-range dependencies.The recurrent layers address the temporal information ignored by independently pooling frame-level predictions.
- Body Key-Points: 137 body key-points, including hand and face points, support the skeleton-based baseline.The input includes the original key-points and their per-frame differences in the extended HCN architecture.
- 3D-CNN: I3D uses an inflated Inception-V1 architecture with successive 3D convolutional and max-pooling layers.C3D did not converge in the reported experiments, so the authors adopted I3D and its suggested implementation details.
5 Experimental Result
The experiments establish signer-independent baselines and examine confusion, language ambiguity, class count, pretraining, and training-sample frequency. Results indicate that errors reflect both prediction difficulty and ambiguity, while more in-domain data and pretraining improve performance.
- 5 Experimental Result: Signer-independent tests use unseen subjects and report average per class accuracy alongside average per class top-five accuracy.These results provide baselines for 2D-CNN, 3D-CNN, CNN-LSTM-HMM, and body key-point approaches.
- 5.1 Qualitative Discussion: Most I3D confusion-matrix mass lies on the diagonal, but several errors exceed .25 for semantically or articulatorily similar signs.Examples include Good→Thanks (.4), Water→Mother (.33), and Today→Now (.33).
- 5.1 Qualitative Discussion: Language ambiguity contributes to model error, especially for classes with more samples, motivating average per class top-five accuracy as a second metric.For smaller training sets, the authors attribute error mainly to prediction errors; with more samples, ambiguity can become more prominent.
- 5.2 The Effect of Pre-Trained Model: 85.32% average per class accuracy and 96.53% average per class top-five accuracy result when ASL100 is initialized from an ASL200-trained I3D model.The authors report more than 3.5% performance boost over the ImageNet+Kinetics-pretrained setting and attribute the result to in-domain pretraining.
- 5.3 The Effect of Number of Classes: 83.36% to 81.97% average per class accuracy follows doubling test classes, while 85.32% to 83.36% follows doubling train classes.The reported effect is significantly smaller when classes have more video samples per class.
- 5.4 The Effect of Number of Video Samples: Accuracy correlates directly with training-video count below 40 samples per class, whereas sign difficulty may matter more above 40 samples.Classes with more than 40 training samples average 80% accuracy, but the authors caution that 40 is not established as a sweet spot.
6 Conclusion
The paper introduces a large-scale, signer-independent ASL dataset spanning 1,000 signs in challenging conditions, benchmarks recognition methods, and identifies I3D as substantially stronger than prior state-of-the-art approaches. It also outlines future directions involving optical flow, body key-points, and segmentation.
- 222 signers and signer-independent sets make the dataset suitable for evaluating generalization across individuals.
- 1,000 signs were recorded under challenging and unconstrained conditions, expanding evaluation beyond limited-vocabulary settings.
- I3D outperformed current state-of-the-art methods by a large margin on the proposed dataset.
- Future work proposes optical flow, body key-points, and training-only segmentation as additional avenues for recognition.