PaperlayerView exact revision on arXiv

Source-linked AI summary

MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language

Hamid Reza Vaezi Joze, Oscar Koller

arXiv:1812.01053v2cs.CV

TL;DR

Sign language recognition lacks large, realistic, labeled datasets suitable for modern deep learning, despite its multimodal and asynchronous nature. The paper introduces MS-ASL with signer-independent evaluation and proposes I3D for recognition, reporting a large-margin advantage over prior state-of-the-art methods.

  • Problem

    Public large-scale ASL resources suitable for evaluating modern deep learning approaches are scarce, while recognition requires substantial labeled data.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs MS-ASL from realistic public videos, provides signer-independent subsets, evaluates baseline methods, and applies I3D to sign recognition.

    Source: §1 · p. 2, Proposed ASL Data Set · p. 4, §4 · p. 8

  • Results

    I3D outperforms current state-of-the-art methods by a large margin on the proposed benchmark.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    MS-ASL provides a large, challenging benchmark for studying signer-independent ASL recognition in realistic recording conditions.

    Source: §6 · p. 10

    The dataset includes repetitive signs, dialectal variation, signer independence, and substantial visual variability across recordings.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Sign language recognition is a challenging and often underestimated problem comprising multi-modal articulators (handshape, orientation, movement, upper body and face) that integrate asynchronously on multiple streams. Learning powerful statistical models in such a scenario requires much data, particularly to apply recent advances of the field. However, labeled data is a scarce resource for sign language due to the enormous cost of transcribing these unwritten languages. We propose the first real-life large-scale sign language data set comprising over 25,000 annotated videos, which we thoroughly evaluate with state-of-the-art methods from sign and related action recognition. Unlike the current state-of-the-art, the data set allows to investigate the generalization to unseen individuals (signer-independent test) in a realistic setting with over 200 signers. Previous work mostly deals with limited vocabulary tasks, while here, we cover a large class count of 1000 signs in challenging and unconstrained real-life recording conditions. We further propose I3D, known from video classifications, as a powerful and suitable architecture for sign language recognition, outperforming the current state-of-the-art by a large margin. The data set is publicly available to the community.

1 Introduction

Sign language recognition needs substantial labeled video data for vision-based models, but public large-scale ASL resources are scarce. MS-ASL addresses this gap with a broad, signer-independent benchmark and an I3D-based approach.

Motivation

  • Vision-based sign recognition avoids specialized cameras, sensors, and gloves, enabling more general usage.Deep learning has also increased the amount of training data needed for strong models.Source: §1 · p. 1

Research gap

  • Public large-scale ASL resources were insufficient for evaluating recent deep learning approaches.The paper frames this shortage as a central barrier to applying current computer vision methods.Source: §1 · p. 2

Contributions

  • MS-ASL provides over 200 signers, signer-independent sets, 1000 signs, and challenging unconstrained recording conditions.The dataset is designed to support realistic generalization studies.Source: §1 · p. 2
  • I3D is proposed as a suitable sign-language-recognition architecture and is reported to outperform previous state-of-the-art methods by a large margin.The paper also evaluates several existing approaches as baselines.Source: §1 · p. 2

2 Previous Works

Earlier sign-language datasets and recognition systems often relied on constrained sensing, limited vocabularies, few signers, or non-reproducible resources. The paper positions MS-ASL as a larger, publicly available benchmark for signer-independent recognition in unconstrained settings.

Recognition methods

  • Many earlier recognition systems used sensor gloves, depth cameras, multiple cameras, colored gloves, or other specialized equipment.These setups constrain applicability compared with single-camera vision-based recognition.Source: §2 · p. 2

Sign language datasets

  • Existing corpora covered vocabularies ranging from 50 to 483 signs, often with few signers or controlled recording conditions.Examples include Purdue RVL-SLLL, RWTH-BOSTON, and SIGNUM resources.Source: §2 · pp. 2–3
  • Some multilingual corpora were not well curated or consistently defined for reproducible research, limiting fair comparisons.The GSL Lemmas Corpus was reported with different vocabulary counts across studies.Source: §2 · p. 3

Benchmark gap

  • RWTH-PHOENIX-Weather 2014 and DEVISIGN were identified as publicly available datasets large enough for recent deep learning, but lacking signer variety and natural recording challenges.MS-ASL is introduced to address these limitations for ASL.Source: §2 · p. 3
  • MS-ASL contributes a publicly available ASL benchmark covering over 200 signers, signer-independent sets, and unconstrained recordings.This directly targets limitations identified in prior datasets.Source: §2 · p. 3

Proposed ASL Data Set

MS-ASL is constructed from realistic public ASL videos and organized into signer-independent subsets spanning 100 to 1000 signs. Its natural variation and ambiguity make it a challenging benchmark, evaluated with class-balanced top-one and top-five accuracy.

Data sources

  • Public ASL videos provide variation in viewpoint, background, lighting, positioning, dialect, and signer identity.These properties make the sources realistic but challenging for dataset construction.Source: Proposed ASL Data Set · p. 4

Annotation

  • Labels were obtained from OCR, video captions, descriptions, and titles, with different procedures for longer and shorter videos.The pipeline used printed labels, temporal captions, and titles as available.Source: Proposed ASL Data Set · p. 4

Signer processing

  • Bounding-box detection and face recognition tracked signers, filtered static-image descriptions, and split videos containing sequential signers.Signer tracking supported construction of signer-independent partitions.Source: Proposed ASL Data Set · p. 4

Dataset organization

  • The dataset releases ASL100, ASL200, ASL500, and ASL1000 subsets covering the 100, 200, 500, and 1000 most frequent words.Each subset has train, test, and validation sets with signer-independent splits.Source: §3.4 · pp. 5–6

Dataset challenges

  • The benchmark includes repetitive performances, dialectal variants, signer independence, and visual variation in backgrounds, lighting, clothing, and camera viewpoint.These factors make it more challenging than conventional video-classification datasets.Source: §3.4 · p. 6

Evaluation

  • Average per class accuracy and average per class top-five accuracy address class imbalance and contextual ambiguity in ASL.Top-five accuracy counts a prediction as correct when the ground-truth label appears among the five guesses.Source: §3.5 · p. 6

4 Baseline Methods

The baseline methods treat isolated sign language recognition as video classification and cover 2D recurrent, body key-point, and 3D convolutional approaches. Experiments use signer-centered video inputs and compare these methods on the proposed ASL data sets.

Baseline Methods

  • Three baseline families use 2D convolution with recurrence, body key-points, or 3D convolution for isolated sign recognition.The paper frames isolated sign recognition as a video classification task and implements at least one method from each action-recognition category.Source: §4 · p. 6

Implementation

  • Inputs are signer bounding boxes resized to 224×224 pixels, with 64-frame temporal windows and consistent horizontal flipping during augmentation.Bounding boxes are extracted with SSD; shorter videos may be elongated by repeating their first or last frame.Source: §4 · pp. 6–7

2D-CNN

  • The 2D-CNN baseline extracts features from each video frame and uses LSTM layers to encode temporal ordering and long-range dependencies.The recurrent layers address the temporal information ignored by independently pooling frame-level predictions.Source: §4 · p. 7

Body Key-Points

  • 137 body key-points, including hand and face points, support the skeleton-based baseline.The input includes the original key-points and their per-frame differences in the extended HCN architecture.Source: §4 · p. 7

3D-CNN

  • I3D uses an inflated Inception-V1 architecture with successive 3D convolutional and max-pooling layers.C3D did not converge in the reported experiments, so the authors adopted I3D and its suggested implementation details.Source: §4 · pp. 7–8

5 Experimental Result

The experiments establish signer-independent baselines and examine confusion, language ambiguity, class count, pretraining, and training-sample frequency. Results indicate that errors reflect both prediction difficulty and ambiguity, while more in-domain data and pretraining improve performance.

5 Experimental Result

  • Signer-independent tests use unseen subjects and report average per class accuracy alongside average per class top-five accuracy.These results provide baselines for 2D-CNN, 3D-CNN, CNN-LSTM-HMM, and body key-point approaches.Source: §5 · p. 8

5.1 Qualitative Discussion

  • Most I3D confusion-matrix mass lies on the diagonal, but several errors exceed .25 for semantically or articulatorily similar signs.Examples include Good→Thanks (.4), Water→Mother (.33), and Today→Now (.33).Source: §5.1 · p. 9
  • Language ambiguity contributes to model error, especially for classes with more samples, motivating average per class top-five accuracy as a second metric.For smaller training sets, the authors attribute error mainly to prediction errors; with more samples, ambiguity can become more prominent.Source: §5.1 · p. 9

5.2 The Effect of Pre-Trained Model

  • 85.32% average per class accuracy and 96.53% average per class top-five accuracy result when ASL100 is initialized from an ASL200-trained I3D model.The authors report more than 3.5% performance boost over the ImageNet+Kinetics-pretrained setting and attribute the result to in-domain pretraining.Source: §5.2 · p. 9

5.3 The Effect of Number of Classes

  • 83.36% to 81.97% average per class accuracy follows doubling test classes, while 85.32% to 83.36% follows doubling train classes.The reported effect is significantly smaller when classes have more video samples per class.Source: §5.3 · p. 10

5.4 The Effect of Number of Video Samples

  • Accuracy correlates directly with training-video count below 40 samples per class, whereas sign difficulty may matter more above 40 samples.Classes with more than 40 training samples average 80% accuracy, but the authors caution that 40 is not established as a sweet spot.Source: §5.4 · p. 10

6 Conclusion

The paper introduces a large-scale, signer-independent ASL dataset spanning 1,000 signs in challenging conditions, benchmarks recognition methods, and identifies I3D as substantially stronger than prior state-of-the-art approaches. It also outlines future directions involving optical flow, body key-points, and segmentation.

  • 222 signers and signer-independent sets make the dataset suitable for evaluating generalization across individuals.Source: §6 · p. 10
  • 1,000 signs were recorded under challenging and unconstrained conditions, expanding evaluation beyond limited-vocabulary settings.Source: §6 · p. 10
  • I3D outperformed current state-of-the-art methods by a large margin on the proposed dataset.Source: §6 · p. 10
  • Future work proposes optical flow, body key-points, and training-only segmentation as additional avenues for recognition.Source: §6 · p. 10

Related papers

Loading 1812.01053v2…