PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

Prem Selvaraj, Gokul NC, Pratyush Kumar, Mitesh Khapra

arXiv:2110.05877v1cs.CLcs.AIcs.CVcs.LG

TL;DR

Sign-language recognition research has lagged NLP, especially for recognizing individual signs and complete sentences. OpenHands releases pose-based datasets and models and evaluates self-supervised pretraining. The paper reports improved recognition, particularly in low-resource settings, and crosslingual transfer from Indian Sign Language.

  • Problem

    Sign-language recognition research lags NLP, with work concentrated on a few languages and few standardized baselines despite limited labelled data across many sign languages.

    Source: §1 · p. 1

  • Method

    OpenHands standardizes pose-based datasets and models across sign languages and evaluates self-supervised pretraining on a curated Indian Sign Language corpus.

    Source: §1 · pp. 1–2

  • Results

    Graph-based pose models achieve state-of-the-art results on most sign languages, and self-supervised pretraining improves fine-tuning, especially in low-resource settings, with crosslingual transfer from Indian Sign Language.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The authors conclude that pose-based models suit real-time applications, and that standardization can help accelerate dataset collection and model benchmarking.

    Source: §1 · p. 2

    The authors leave specially designed sign-language-domain representations for future work, including alternatives to directly posing an interpolation-like task for masked tokens.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

AI technologies for Natural Languages have made tremendous progress recently. However, commensurate progress has not been made on Sign Languages, in particular, in recognizing signs as individual words or as complete sentences. We introduce OpenHands, a library where we take four key ideas from the NLP community for low-resource languages and apply them to sign languages for word-level recognition. First, we propose using pose extracted through pretrained models as the standard modality of data to reduce training time and enable efficient inference, and we release standardized pose datasets for 6 different sign languages - American, Argentinian, Chinese, Greek, Indian, and Turkish. Second, we train and release checkpoints of 4 pose-based isolated sign language recognition models across all 6 languages, providing baselines and ready checkpoints for deployment. Third, to address the lack of labelled data, we propose self-supervised pretraining on unlabelled data. We curate and release the largest pose-based pretraining dataset on Indian Sign Language (Indian-SL). Fourth, we compare different pretraining strategies and for the first time establish that pretraining is effective for sign language recognition by demonstrating (a) improved fine-tuning performance especially in low-resource settings, and (b) high crosslingual transfer from Indian-SL to few other sign languages. We open-source all models and datasets in OpenHands with a hope that it makes research in sign languages more accessible, available here at https://github.com/AI4Bharat/OpenHands .

1 Introduction

OpenHands addresses lagging sign-language recognition research by bringing low-resource NLP strategies to sign languages and releasing standardized datasets and models. It reports benefits from pose-based recognition and self-supervised pretraining, including crosslingual transfer.

Motivation

  • Sign-language recognition research has lagged NLP, with work concentrated on a few languages and few standardized baselines, despite limited labelled data across many languages.The paper distinguishes recognition of individual signs from recognition of complete sentences.Source: §1 · p. 1

Contributions

  • OpenHands standardizes pose-based data for six sign languages and releases 28 trained models across seven datasets, with deployment scripts for CPU and GPU use.Pose extractors convert RGB video into frame-wise keypoint coordinates; the authors report real-time performance for the released models.Source: §1 · pp. 1–2
  • The authors curate 1,129 hours of Indian Sign Language video for self-supervised pretraining and report improved fine-tuning, especially in low-resource settings, plus crosslingual transfer.The reported transfer is from Indian Sign Language to other sign languages; the paper releases pretrained and fine-tuned models for four languages.Source: §1 · p. 2

Results

  • Graph-based pose models achieve state-of-the-art results on most sign languages, while self-supervised pretraining improves low-resource recognition and transfers across several languages.LSTM models are faster and may suit constrained devices; pose-based pipelines are described as efficient for real-time use.Source: §1 · p. 2

Contribution

  • OpenHands releases its models and datasets to make sign-language recognition research more accessible and standardized, and invites contributions representing more languages.The paper also identifies standardized datasets and models as a way to help accelerate dataset collection and model benchmarking.Source: §1 · p. 2

2 Background and Related Work

The background defines sign-language recognition tasks, reviews pose-based sequence and graph models, and surveys pretraining strategies that could apply to sign-language recognition.

2 Background and Related Work

  • Prior ISLR research has produced datasets and deep-learning architectures, with this review emphasizing pose-based models.Source: §2 · p. 2

2.1 Sign Language

  • Sign languages use bodily actions to communicate; ISLR maps each visual sign to a gloss, whereas CSLR maps continuous signs to serialized glosses.The paper also distinguishes sign-language translation, which converts signs into spoken-language text.Source: §2.1 · p. 2

2.2 Models for ISLR

  • Pose-based ISLR uses frame-wise keypoints; sequence models classify temporal pose sequences, while graph models encode skeleton joints and temporal connections.Prior work includes ST-GCN and SL-GCN; SL-GCN combines attention with Decoupled-GCN and is reported as achieving state-of-the-art results.Source: §2.2 · p. 3

2.3 Pretraining strategies

3 Standardized Pose-based ISLR Models across Sign Languages

The section standardizes pose-based ISLR data and model comparisons across seven datasets, then evaluates accuracy, inference latency, and training cost to inform model selection.

3.1 ISLR Datasets

  • Seven datasets across six sign languages are converted into pose inputs using 27 selected 2D keypoints covering the upper body, hands, and face.MediaPipe Holistic returns 75 keypoints; the input has F × K × D dimensions, with K=27 and D=2.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Standardized ISLR Models

  • Four standardized models—LSTM, BERT, ST-GCN, and SL-GCN—are trained with similar setups to make comparisons across datasets more consistent.The architectures include sequence-based RNN and Transformer models and graph-based spatio-temporal GCN models.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 Experimental Setup and Results

  • Graph-based models achieve state-of-the-art accuracy across all seven datasets, improving on published results on six except AUTSL.The reported improvement exceeds 10% on GSL; Table 2 gives the cross-dataset accuracy comparison.Source: Table 2 · p. 5, §3.3 · p. 5
  • LSTM is an order of magnitude faster than SL-GCN but costs about 10% accuracy, while ST-GCN is about 2× faster than SL-GCN with only 3% lower average accuracy.SL-GCN takes 4× longer to train than ST-GCN; the authors select ST-GCN for later pretraining, balancing accuracy, latency, and training cost.Source: §3.3 · p. 5

4 Self-Supervised Learning for ISLR

The paper builds a large Indian Sign Language pose corpus and compares self-supervised pretraining strategies for isolated sign recognition. Dense Predictive Coding (DPC) is the effective strategy reported, with benefits for low-resource in-language and crosslingual fine-tuning.

4.1 Indian SL Corpus for Self-supervised pretraining

  • The processed Indian-SL corpus retains 1,129 hours from over 1,500 downloaded hours, yielding more than 100 million 75-keypoint pose frames in an open 250 GB dataset.Videos with noisy pose data or no person or multiple people were removed.Source: §4.1 · pp. 5–6

4.2 Pretraining Setup and Experiments

  • Masked regression reduces pretraining loss but gives no major INCLUDE accuracy gain; interpolation fits masked frames, while prediction errors diverge as masking spans lengthen.Experiments with 20% and 30% masking ratios and contiguous spans of 2–10 frames produced similar results.Source: §4.2 · pp. 6–7
  • Direction-loss pretraining does not converge, which the authors attribute to noisy quadrant labels from pose jitter and a four-direction target that omits static motion vectors.The authors report that static vectors account for more than half of the total motion vectors.Source: §4.2 · p. 7
  • Contrastive pretraining converges on InfoNCE loss but yields no INCLUDE gain over scratch training, and its embeddings do not cluster videos by sign class.Different augmentations of a clip have similar embeddings, but this does not produce class-discriminative representations.Source: §4.2 · pp. 7–8
  • DPC fine-tuning improves INCLUDE accuracy by 3.5% over training from scratch; the authors attribute its success to coarse-grained representations that capture motion semantics across frames.The model predicts embeddings for three future windows from four input windows.Source: Predictive-coding based · p. 8

4.3 Evaluation on low-resource and crosslingual settings

5 The OpenHandslibrary

OpenHands packages pose-based datasets, models, and deployment utilities in a modular library intended to make sign-language research easier to extend and use. It also documents inference trade-offs and pose-processing transforms for training and deployment.

5 The OpenHandslibrary

  • The library releases pose-based datasets for six sign languages, four ISLR models trained on seven datasets, an Indian-SL pretraining corpus with over 1,100 hours of pose data, and pretrained and fine-tuned models with deployment scripts.The deployment scripts use MediaPipe pose estimation and the trained ISLR models.Source: §5 · p. 9

5.1 Inference Benchmarking

  • Inference benchmarking uses serial batch-size-1 processing on INCLUDE videos, while the pose-estimation variants trade accuracy for speed and the authors prefer lite or full for real-time use depending on the CPU.The heavy variant is used for training and testing; the authors say it is suitable for real-time use only with frame-skipping and decoders below 8 fps.Source: ISLR Model Inference · p. 9, MediaPipe Inference · p. 9

5.2 Pose Transforms

  • Pose transforms normalize video coordinates and body-centered scale, interpolate missing or corrupt keypoints, and augment poses to model viewpoint, scale, timing, and segmentation variation.The library also supports uniform or random temporal subsampling when clips exceed a frame limit.Source: Pre-processing · p. 10, Augmentations · p. 10

5.3 Library Structure

  • The library uses configurable encoder-decoder models and extensible base classes for datasets, allowing new encoders, decoders, and datasets to be added with limited changes.Beginners can run workflows through configuration files, while researchers can import and customize modules.Source: §5.3 · p. 10

6 Conclusion and Future work

OpenHands releases pose-based ISLR datasets and models across sign languages, identifies graph-based methods such as ST-GCN as accurate and efficient, and finds DPC pretraining effective for in-language and crosslingual transfer. The authors outline future work on representations, datasets, models, and deployment.

6 Conclusion and Future work

  • The authors release pose-based datasets and four ISLR models across six sign languages, identifying graph-based methods such as ST-GCN as accurate and efficient.They open-source models, datasets, training, and deployment scripts through OpenHands.Source: §6 · pp. 10–11
  • DPC is reported as effective for pretraining, with benefits for both in-language and crosslingual transfer.The conclusion also identifies future work on face landmarks, pose estimation, alternative graph models, data sampling, quantized inference, low-FPS videos, other pretraining languages, CSLR, and deployment.Source: §6 · p. 11

Related papers

Loading 2110.05877v1…