Source-linked AI summary
How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language
Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, Xavier Giro-i-Nieto
TL;DR
Sign language processing lacks large-scale annotated datasets, especially for continuous sign language. How2Sign addresses this gap with a multimodal, multiview ASL dataset and evaluates generated videos with native signers, who could understand them to a certain degree.
Problem
Large-scale annotated datasets for continuous sign language are scarce, although real-world sign language processing commonly involves complete sentences.
Method
How2Sign provides over 80 hours of multiview, multimodal ASL video with aligned transcripts, speech, depth, glosses, and pose annotations, including a three-hour Panoptic subset for 3D keypoints.
Results
Native signers verified that How2Sign’s 2D keypoints were sufficient to a certain degree for classifying and transcribing ASL videos into English.
Takeaways & Limitations
How2Sign supports sign language recognition, translation, production, and 3D human pose estimation, while connecting vision, speech, and language research.
Takeaways & Limitations
Pose-based visualizations were insufficient for complete sentence understanding because current pose estimation lacked precision for fast hand movements.
Abstract
from arXiv · showhide
One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth. A three-hour subset was further recorded in the Panoptic studio enabling detailed 3D pose estimation. To evaluate the potential of How2Sign for real-world impact, we conduct a study with ASL signers and show that synthesized videos using our dataset can indeed be understood. The study further gives insights on challenges that computer vision should address in order to make progress in this field. Dataset website: http://how2sign.github.io/
1. Introduction
Sign language technologies face communication barriers because sign languages differ from spoken and written languages, while large annotated continuous ASL datasets remain scarce. How2Sign addresses this gap with a large multimodal, multiview dataset and evaluates whether generated videos are understandable to ASL signers.
- Motivation: 466 million deaf or hard-of-hearing people worldwide use sign languages as their primary communication means, yet most technologies support spoken or written language instead.Sign languages are evolving natural languages with linguistic structure, and ASL is not a signed form of English.
- Research gap: Large-scale annotated datasets for continuous sign language remain very limited, and no ASL dataset was large enough for recent deep learning approaches.Continuous datasets require sentence-level segmentation and annotation.
- Dataset: The dataset aligns more than 35k sentences and includes a 16k-word English vocabulary, automatically extracted 2D keypoints for more than 6M frames, and a three-hour Panoptic subset with 3D keypoints.The Panoptic studio used hundreds of cameras and sensors for detailed 3D reconstruction and pose estimation.
- Evaluation: A study with ASL signers found that videos generated using How2Sign could be understood to a certain extent and revealed challenges for computer vision.The study evaluates potential real-world impact rather than only dataset scale.
2. Background and Related Work
Sign language processing requires large continuous datasets because meaning depends on manual and non-manual features, grammatical structure, transitions, and non-monotonic alignment. Existing datasets are often limited in vocabulary, duration, or domain, whereas How2Sign broadens coverage with a large ASL corpus and parallel modalities.
- Challenges in sign language processing: Sign language processing must represent handshape, orientation, movement, location, facial expressions, head motion, mouth movements, and eye gaze.These manual and non-manual features jointly convey meaning.
- Challenges in sign language processing: Concatenating isolated signs cannot capture complete continuous-language sentences because sign transitions, grammatical structures, and non-monotonic sign-to-speech alignment matter.A sign’s beginning or ending may change depending on neighboring signs.
- Dataset gap: Large-scale annotated datasets are absent, while many existing resources contain isolated signs even though real-world applications commonly involve complete sentences.Isolated-sign datasets remain useful for dictionaries and language learning.
- Existing datasets: Public continuous sign language datasets are constrained by limited vocabulary, short duration, or narrow domains, and collecting and annotating them is laborious and expensive.RWTH-Phoenix-2014T, for example, contains 11 hours of weather-broadcast data in one domain.
- How2Sign: How2Sign provides a considerably larger vocabulary in the continuous setting across a broader discourse domain and is the first sign language dataset containing speech through alignment with How2.The dataset also offers the most parallel modalities among the compared resources, according to Table 1.
3. The How2Sign dataset
How2Sign is a large-scale, multimodal, multiview ASL dataset built from instructional videos, with aligned clips, linguistic annotations, pose information, and multiple recording views. Its design supports multimodal learning and analysis of signer generalization while documenting important privacy and demographic limitations.
- Dataset composition: How2Sign pairs approximately 80 hours of ASL video with speech, transcriptions, gloss annotations, categorization, and other modalities.The dataset combines existing English translations and speech with collected or automatically extracted modalities.
- Recording setup: All 80 hours were recorded from multiple angles, while the approximately three-hour Panoptic subset used more than 500 cameras for high-quality 3D keypoint estimation.Multiple views make signs more visible and reduce hand occlusion and ambiguity.
- Annotations: The dataset includes automatically extracted 2D body, face, and hand keypoints and manually collected gloss and sentence-boundary annotations.Green Screen recordings provide pose information for frontal and side views, while gloss annotations are time-aligned through ELAN.
- Dataset statistics: The recordings cover more than 35k sentence-level clips and a vocabulary exceeding 16k English words.Each clip averages 162 frames, or 5.4 seconds, and 17 words.
- Dataset statistics: How2Sign includes a 26-video test subset recorded by a signer absent from training for measuring generalization across signers.The recordings involve 11 signers distributed across dataset splits.
- Privacy and bias: The dataset’s participants and recording conditions limit diversity in language variety, race or ethnicity, skin tone, scenery, lighting, and camera quality.Participants were born and raised in the United States, and signers may have monitored their production more carefully because their faces were recorded.
4. Evaluating the potential of How2Sign for sign language tasks
The study evaluates whether keypoints from How2Sign can support understandable synthesized ASL videos, using skeleton visualizations and GAN-generated videos. GAN-generated videos outperform skeletons across reported metrics, but both remain insufficient for fine-grained sentence understanding because hands and facial expressions are inadequately represented.
- 4.1. Synthesizing sign language videos: Keypoints from How2Sign were used to create skeleton visualizations and GAN-generated videos for evaluating sign-language understanding.The GAN approach synthesizes signer videos from predicted keypoints and separately models facial landmarks.
- 4.2. Can ASL signers understand generated sign language videos?: The evaluation measured category classification, perceived understanding, and English translation quality with four ASL signers.Two signers watched skeleton visualizations and two watched GAN-generated videos; results used Accuracy, MOS, and BLEU.
- 4.2. Can ASL signers understand generated sign language videos?: GAN-generated videos achieved higher scores than skeleton visualizations across classification, MOS, and BLEU.The comparison is reported over all three evaluation metrics.
- 4.2. Can ASL signers understand generated sign language videos?: Both visualization types supported mostly correct classification of video topics.This indicates topic-level understanding was possible for both skeleton and GAN-generated representations.
- 4.2. Can ASL signers understand generated sign language videos?: Neither skeletons nor GAN-generated videos conveyed enough information for complete sentence-level English translation.The limitation is attributed to missing or inaccurate hand movements and detailed facial expressions, which are central to ASL understanding.
5. Conclusion
How2Sign is a large-scale multimodal and multiview ASL dataset designed to support sign-language understanding and related multimodal research. A signer study found that keypoint-based synthetic videos were understandable to a certain extent while exposing unresolved generation challenges.
- 5. Conclusion: How2Sign provides more than 80 hours of ASL videos with speech, English transcripts, and annotations for recognition, translation, production, and related multimodal tasks.The dataset extends How2 with an ASL modality and connects vision, speech, and language research.
- 5. Conclusion: The authors studied whether ASL signers could understand videos generated from automatically extracted dataset annotations.The conclusion describes this as the first study of understanding keypoint-based synthetic videos by sign-language users.
- 5. Conclusion: The study indicates that current video synthesis methods support understanding to a certain extent but do not fully resolve sign-language video generation.The broader evaluation identifies hands and facial expressions as important remaining challenges.
6. Sign Language
ASL conveys meaning through manual features and non-manual markers, including facial and head movements. These features must be considered together to capture complete sign meaning, making sign-language processing challenging for computer vision.
- 6. Sign Language: Sign languages combine manual features such as handshape, palm orientation, movement, and location with non-manual markers.Non-manual markers include head, mouth, eyebrow, cheek, facial-expression, and eye-gaze movements.
- 6. Sign Language: Recognizing, translating, or generating signs requires accounting for both feature types to capture complete meaning.The passage identifies this multimodal linguistic structure as a challenge for computer vision.
7. How2Sign dataset
How2Sign combines multimodal annotations, signer metadata, and studio recordings with analyses of pose quality and generalization. The dataset also documents linguistic and signing variation across its contributors.
- Annotations: How2Sign includes gloss annotations collected with ELAN and conventions for representing signs, compounds, classifiers, locations, and pointing.Glosses approximate ASL signs rather than translating their meanings directly.
- Signers and recording settings: The dataset contains recordings from 11 signers across Green Screen and Panoptic studios, with signer IDs provided for each video.Four signers participated in both studios; others participated in only one recording location.
- Keypoint quality: 53.4% average keypoint confidence at 1280 x 720 exceeded 42.4% at 210 x 260.Both hands were especially harmed by low-resolution input.
- Generalization: The dataset supports evaluation of generalization to 413 validation and 510 test out-of-vocabulary words.These words occur in validation or test but not in training.
- Generalization: Test recordings by a signer absent from training support evaluation of cross-signer generalization to out-of-distribution subjects.This setup can help assess how well models recognize or translate signs from unseen signers.
- Language and signing variation: Examples document language-register variation and intra-sign variation, including different expressions of “I am” and different pointing locations for “hair.”The corresponding gloss for the latter example is IX-LOC-HAIR.