Source-linked AI summary
Quantitative Survey of the State of the Art in Sign Language Recognition
Oscar Koller
TL;DR
Sign language recognition research has attracted increasing attention, but a quantitative overview of the field and its findings is needed. This paper manually categorizes studies and analyzes the PHOENIX benchmark literature; it finds that non-manual parameters remain rare and PHOENIX is the only worldwide resource for large-vocabulary continuous recognition.
Problem
The field's growing number of studies and datasets calls for a quantitative overview of sign language recognition research.
Source: §1 · p. 1Method
The paper manually tags over 300 studies and synthesizes their characteristics, including a structured analysis of PHOENIX benchmark studies.
Source: §1 · p. 1, §4 · p. 12Results
Non-manual parameters remain rare, and RWTH-PHOENIX-Weather, with 1080 signs, is the only worldwide resource for large-vocabulary continuous sign language recognition.
Source: §4 · p. 13Takeaways & Limitations
The authors call for publicly accessible real-life large-vocabulary continuous tasks with well-defined train, development, and test partitions.
Source: §4 · p. 13Many papers do not clearly describe augmentation, and no study has analyzed the effects of different augmentation options.
Source: §3 · p. 11
Abstract
from arXiv · showhide
This work presents a meta study covering around 300 published sign language recognition papers with over 400 experimental results. It includes most papers between the start of the field in 1983 and 2020. Additionally, it covers a fine-grained analysis on over 25 studies that have compared their recognition approaches on RWTH-PHOENIX-Weather 2014, the standard benchmark task of the field. Research in the domain of sign language recognition has progressed significantly in the last decade, reaching a point where the task attracts much more attention than ever before. This study compiles the state of the art in a concise way to help advance the field and reveal open questions. Moreover, all of this meta study's source data is made public, easing future work with it and further expansion. The analyzed papers have been manually labeled with a set of categories. The data reveals many insights, such as, among others, shifts in the field from intrusive to non-intrusive capturing, from local to global features and the lack of non-manual parameters included in medium and larger vocabulary recognition systems. Surprisingly, RWTH-PHOENIX-Weather with a vocabulary of 1080 signs represents the only resource for large vocabulary continuous sign language recognition benchmarking world wide.
1 Introduction
This work provides a quantitative meta-study of sign language recognition, manually labeling 300 relevant studies and analyzing their characteristics. It also offers a detailed comparison of studies on the PHOENIX 2014 benchmark and makes its source data public.
1 Introduction
- The study manually labels 300 recognition studies by vocabulary, signers, dataset quality, input data, and modeled sign-language modalities and features.It presents analyses using graphics and tables that relate characteristics, visualize correlations, and highlight shortcomings.Source: §1 · p. 1
- The paper makes extensive quantitative structured data on sign language recognition research publicly available.The full data table is in the appendix.Source: §1 · p. 2
- It presents the first sign language recognition meta-study, providing quantitative insights and analysis of the state of the art.Source: §1 · p. 2
- It provides the first overview and in-depth analysis of published papers comparing recognition systems on the PHOENIX 2014 standard benchmark.The analysis tracks architectures, training styles, losses, and data augmentation alongside performance.Source: §1 · pp. 1–2
2 Analysis of the State of the Art
Across sign language recognition research, experiments grew over time while input practices shifted toward non-intrusive capture and global features. The analysis also finds that large-vocabulary continuous recognition is concentrated on a single benchmark resource.
2 Analysis of the State of the Art
- Isolated-recognition publications grew at an apparently exponential rate, while continuous-recognition publications grew close to linearly; isolated studies averaged at least twice as many.The authors suggest this pattern may reflect the difficulty of continuous recognition and scarcity of available training corpora.Source: §2 · p. 2
2.1 Type of Employed Input Data
- RGB is the most popular input across vocabulary sizes; colored gloves appear only in small and medium tasks, while input mixes vary by sign language.ASL results are dominated by RGB; CSL results most often use RGB-D or RGB, and DGS and most other languages focus mainly on RGB.Source: Table 2 · p. 4, §2.1 · p. 4, Table 4 · p. 5
- After 2005, intrusive capturing methods’ prevalence decreased from around 70% to less than 30%, while they appeared in about one quarter of experiments with vocabularies up to 500 signs.Source: §2.1 · p. 4
2.2 Modeled Sign Language Parameters
- Hand shape is the most covered parameter below 1,000 signs, while full-frame features lead above 1,000 and became the most frequent feature after 2015.Full-frame features alone are the most common combination in tasks above 1,000 signs; hand shape is preferred below 50 signs.Source: §2.2 · pp. 5–6, Table 5 · p. 6, Table 7 · p. 7
2.3 Analysis by Sign Language
- ASL has the most results below 200 signs, CSL leads at larger vocabularies, and DGS has significantly more research above 1,000 signs; RWTH-PHOENIX-Weather is the only large-scale continuous resource identified.The paper notes that reproducible research also depends on fixed train, development, and test partitions, accessible downloads, and corpus licenses.Source: §2.3 · pp. 6, 8
2.2 Modeled Sign Language Parameters
- 18% of results with vocabularies up to 50 signs used global features, rising to 62% for vocabularies above 1,000 signs.The analysis associates the growth with body-joint and full-depth availability after Kinect's release and the later shift toward deep learning and full-frame inputs.Source: §2.2 · pp. 5–6, Table 6 · p. 7
2.4 Change of Continuous Recognition Landscape After 2015
- 80 continuous-recognition results appeared before 2015 and 66 after; post-2015 results peak near 30 at 1,080 signs, with a smaller peak at 178.The study attributes the concentration to increased focus on RWTH-PHOENIX-Weather 2014 and CSL.Source: §2.4 · pp. 8, 10, Figure 3 · p. 10
3 Analysis of PHOENIX 2014 Benchmark Papers
The PHOENIX 2014 analysis compares published systems’ features, architectures, training choices, and word error rates. It finds recurring patterns among strong systems, alongside gaps in augmentation reporting and evaluation of its effects.
3 Analysis of PHOENIX 2014 Benchmark Papers
- Most best-performing PHOENIX systems use iterative training; all but Cheng et al. among those below 27.0% WER do so.The survey describes iterative training as using model-generated pseudo-labels to train part or all of the recognition network; some studies report it helps address vanishing gradients in deep CNNs followed by BLSTMs.Source: §3 · p. 10
- Most networks below 30.0% WER use both cross-entropy and CTC losses, while reported loss combinations vary.Other reported terms include KL divergence, squared error, and smooth-L1 loss.Source: §3 · p. 10
- On PHOENIX, all best-performing approaches with WERs below 25.0% employ 2D+1D convolutions [Cheng et al., 2020, Cui et al., 2019, Papastratis et al., 2020, Zhou et al., 2020a]; all recent studies rely on fullframe input images [Adaloglou et al., 2020, Cheng et al., 2020, Papastratis et al., 2020, Zhou et al., 2020b, Zhou et al., 2020a].Source: §3 · p. 10
- Random cropping is the most popular augmentation, but many papers omit augmentation details and no study analyzes the effects of different options.The survey notes that augmentation can significantly affect results.Source: §3 · p. 11
- PHOENIX comparisons use whole-sign units at inference; only two works use subunits, for alignment guidance or pretraining.The cited examples are for alignment and for pretraining.Source: §3 · p. 11
- In the RWTH-Phoenix Weather 2014 comparison, the USTC ST Multi-Cue Network row reports WER values of 21.1 and 20.7; the table covers development and test partition results.Source: Table 11 · p. 11, §3 · p. 11
4 Conclusion and Outlook
The survey synthesizes sign language recognition research through 2020 and identifies trends in task scale, capture methods, and modeled features. It also highlights the scarcity of large-vocabulary continuous benchmarks and gaps in PHOENIX reporting and analysis.
4 Conclusion and Outlook
- The survey analyzes over 300 publications from 1983 to 2020, manually tagging studies with research categories.Its source data is shared in the supplemental materials.Source: §4 · p. 12
- More studies address isolated than continuous recognition, but most isolated studies cover small vocabularies.The conclusion presents this as a field-wide pattern in the surveyed studies.Source: §4 · p. 13
- After 2005, the field shifted from intrusive capture methods toward non-intrusive methods.Source: §4 · p. 13
- Deep learning coincided with predominant use of global, fullframe-based features, especially in larger-vocabulary tasks.Source: §4 · p. 13
- Non-manual parameters remain rare; no surveyed work includes eye gaze or blinks, and DGS alone incorporates them in tasks of at least 200 signs.CSL studies have hardly incorporated non-manual parameters despite CSL being the second most frequently researched sign language.Source: §4 · p. 13
- RWTH-PHOENIX-Weather, with 1080 signs, is the only worldwide resource for large-vocabulary continuous sign language recognition.Source: §4 · p. 13
- On PHOENIX, strong systems typically use iterative training, and most state-of-the-art systems use 2D followed by 1D convolutions on fullframe inputs.Many studies do not describe augmentation carefully, and the effects of different augmentation methods remain unexamined.Source: §4 · p. 13
- The authors call for greater involvement of Deaf researchers and publicly accessible real-life large-vocabulary continuous tasks with defined train, development, and test partitions.Source: §4 · p. 13
Supplemental Material
Supplemental Table S1 catalogs most sign language recognition studies published through early 2020 that report task results or introduce a related corpus. It records key properties of each work, including vocabulary, signers, input data, modeled features, and benchmark status.
Supplemental Material
- Table S1 lists key properties for most sign-language recognition studies through early 2020 that report task results or introduce a corpus.Properties include vocabulary, signer counts, isolated or continuous data, input data, modeled features, and benchmark status.Source: Table S1 · p. 31