PaperlayerView exact revision on arXiv

Source-linked AI summary

Sign Language Production: A Review

Razieh Rastgoo, Kourosh Kiani, Sergio Escalera, Mohammad Sabokrou

arXiv:2103.15910v1cs.CV

TL;DR

Spoken-to-sign and sign-to-spoken translation is needed for mutual communication, yet SLP remains challenging. This survey reviews deep-learning SLP advances and organizes work across modalities, datasets, applications, and models; it reports accurate results while identifying unresolved challenges.

  • Problem

    SLP remains challenging, and the field lacks a detailed review synthesizing recent deep-learning advances, their advantages, limitations, and future directions.

    Source: §1 · p. 1

  • Method

    The survey reviews recent deep-learning SLP work, organizing it by input modalities, datasets, applications, and models.

    Source: §3 · p. 6

  • Results

    The survey reports accurate results from recent SLP models and significant translation results from NMT-based methods, while noting that challenges remain.

    Source: §3 · pp. 6–7

  • Takeaways & Limitations

    NMT-based translation still faces domain adaptation, training-data, uncommon-word, word-alignment, and beam-search challenges.

    Source: §3 · p. 7

    Many proposed SLP models generate only low-resolution samples, while real-world communication requires multi-signer generation with varied appearances and configurations.

    Source: §3 · p. 8

Abstract

from arXiv · show

Sign Language is the dominant yet non-primary form of communication language used in the deaf and hearing-impaired community. To make an easy and mutual communication between the hearing-impaired and the hearing communities, building a robust system capable of translating the spoken language into sign language and vice versa is fundamental. To this end, sign language recognition and production are two necessary parts for making such a two-way system. Sign language recognition and production need to cope with some critical challenges. In this survey, we review recent advances in Sign Language Production (SLP) and related areas using deep learning. This survey aims to briefly summarize recent achievements in SLP, discussing their advantages, limitations, and future directions of research.

1. Introduction

The paper frames sign language production (SLP) as a challenging counterpart to sign-language recognition, needed for two-way communication between hearing and deaf communities. It surveys deep-learning advances and their limitations.

1. Introduction

  • Sign-language recognition has advanced rapidly, while SLP still requires interpreting between visual and linguistic information.Source: §1 · p. 1
  • SLP remains challenging because spoken and sign languages differ in visual form, grammar, tokenization, and word order, making translation more than word-by-word mapping.The survey focuses on deep-learning-based models and reviews their advances and weaknesses.Source: §1 · p. 1
  • The survey's taxonomy summarizes reviewed SLP works, and later sections discuss developments, advantages, limitations, and future research directions.Source: Figure 1 · p. 2, §1 · p. 2

2. SLP Taxonomy

The section organizes deep-learning work in SLP through a taxonomy of its main concepts and research categories.

2. SLP Taxonomy

  • The section presents a taxonomy summarizing the main concepts related to deep learning in SLP.Source: §2 · p. 2
  • The taxonomy categorizes recent SLP works and discusses each category separately.Source: §2 · p. 2
  • Its categories cover input modalities, datasets, applications, and proposed models, with the taxonomy shown in Figure 1.Source: §2 · p. 2

2.1. Input modalities

SLP processes visual and linguistic inputs, with different challenges for images and videos versus text translation. The section outlines common input forms and models, alongside limitations that remain.

2.1. Input modalities

  • Visual input commonly uses RGB images or videos and skeletons; skeletons reduce input dimensions and support lower-complexity, faster models.Source: §2.1 · p. 2
  • RGB videos add temporal complexity, and GANs and LSTMs are commonly used for static and dynamic visual modalities, while more lifelike outputs remain a goal.Source: §2.1 · p. 2
  • Text is the most common linguistic input, and NMT is the most-used text-processing model, but domain adaptation, data quantity, rare words, long sentences, and alignment remain challenges.Transfer learning and byte-pair encoding are described as approaches to domain adaptation and rare-word translation, respectively.Source: §2.1 · p. 2

2.2. Datasets

The section describes the limited public dataset resources for SLP and the scope and collection challenges that constrain evaluation and generalization.

2.2. Datasets

  • Only a few publicly available large-scale SLP datasets exist, and the two widely used benchmarks cover German and American sentences but are insufficient for generalization.Source: §2.2 · pp. 2–3
  • 38611 sequences and 4k vocabularies are reported for How2Sing, while RWTH-PHOENIX-Weather 2014T contains 8257 sequences by 9 signers.The datasets include gloss annotations; RWTH-PHOENIX-Weather 2014T also has 1066 glosses and 2887 spoken-language vocabularies.Source: §2.2 · p. 3
  • Broader SLP datasets need diverse sign languages and paired spoken- and sign-language annotations, but collecting and annotating them is difficult.Source: §2.2 · p. 3
  • Capturing signing requires balancing cost, measurement accuracy, and spontaneity, while synchronizing channels and handling device calibration, noise, and data loss add difficulties.Source: §2.2 · p. 3

2.3. Applications

The section describes tools intended to improve communication access for Deaf people, including machine-translation applications. It notes that real-world accessibility remains limited by restricted vocabularies or sentences, licensing, and narrow application scenarios.

2.3. Applications

  • Interpreters are effective but costly, motivating approaches to inform deaf people when an interpreter is unavailable.The section says these projects aim to provide communication tools for Deaf people, for whom many internet applications are inaccessible.Source: §2.3 · p. 3
  • Deep learning-based Seq2Seq and NMT models have shown promising results in sign language production, but sign language lacks large, diverse corpora.The section describes preprocessing spoken-language text and notes the lack of large, diverse sign-language corpora.Source: §2.3 · p. 3
  • Current applications remain far from supporting large vocabularies and sentences in real-world scenarios, with licensing and limited generalization also identified as challenges.Only some applications are freely available, and current systems are developed for specific application scenarios.Source: §2.3 · p. 3

2.4. Proposed models

The review organizes deep-learning work in sign language production into five model categories and describes their approaches, uses, and remaining challenges. Across the categories, it highlights constraints including data availability, model complexity, and training stability.

2.4. Proposed models

  • The review groups recent SLP models into avatar, NMT, Motion Graph, conditional image/video generation, and other approaches.It presents these categories as a review of recent SLP work and summarizes the reviewed models in Table 1.Source: §2.4 · pp. 3–4

2.4.1 Avatar Approaches

  • Avatar systems use 3D animated models to display signed conversation without corresponding human-signer videos, but unnatural motion and missing nonmanual information can lead to misunderstanding.Although highly realistic results are reported, avatar data collection and annotation restrict results to a small phrase set and require expert checks.Source: §2.4.1 · p. 4

2.4.2 NMT approaches

  • NMT-based SLP can translate spoken sentences into sign pose sequences before generating plausible sign videos, with comparable results to state-of-the-art alternatives on PHOENIX14T.The described pipeline uses NMT, GANs, and motion generation with minimal training-data annotation. NMT challenges include domain adaptation, training-data quantity, uncommon words, alignment, and beam-search settings.Source: §2.4.2 · pp. 4–5

2.4.3 Motion Graph approaches

  • Motion Graphs combine motion-capture data with graph search to generate plausible, controllable motion, but require diverse data and face scalability and computational-complexity challenges.In continuous SLP, sign glosses can be embedded in a graph, with transition probabilities supplied by an NMT decoder at each time step.Source: §2.4.3 · p. 5

2.4.4 Conditional image/video generation

  • SLP video generation must maintain plausible motion between frames while controlling complex human actions and appearances.Reviewed methods include VAE-GAN combinations for sign video and image generation; generative-model training challenges include mode collapse, non-convergence, instability, objective selection, and optimization.Source: §2.4.4 · p. 5

2.4.5 Other models

  • Other SLP models include transformer-based continuous sign generation, skeletal production, text-to-skeletal-pose generation, and multi-channel assessment.The review identifies model complexity and the accuracy-versus-task-complexity trade-off as open issues.Source: §2.4.5 · pp. 5–6

3. Discussion

The survey organizes deep-learning SLP across input modalities, datasets, applications, and model families, finding progress alongside persistent limits in data, realism, and generalization. It identifies research directions and practical requirements for real-world communication.

Input modalities

  • RGB and skeleton inputs support visual SLP, while text is the common linguistic input; GANs/LSTMs and NMT are widely used, but lifelike visuals and translation challenges remain.Skeleton inputs reduce dimensionality and can support faster, lower-complexity models; text translation still faces domain adaptation, uncommon words, alignment, and tokenization challenges.Source: §3 · p. 6

Datasets

  • Large, sentence-level paired datasets remain scarce, and the commonly used RWTH-PHOENIX-Weather 2014T and How2Sign benchmarks cover only German and American sentences.The survey says these benchmarks are not enough to establish model generalization across sign languages.Source: §3 · p. 6

Applications

  • Existing SLP applications can bridge Deaf and hearing communities, but most target specific domains and do not yet cover large vocabularies or complex real-world scenarios.Only some applications are freely available, and licensing is another challenge.Source: §3 · p. 6

Proposed models

  • Reviewed model families include avatar, NMT, motion-generation, conditional image/video-generation, and other approaches; avatars can be realistic but are limited to small phrase sets by costly, unscalable data collection.Figures 3–5 illustrate generated gloss/video, pose-sequence, and signing-avatar outputs.Source: §3 · pp. 6–7, Figure 3 · p. 8, Figure 4 · p. 8
  • NMT systems face domain adaptation, limited training data, uncommon-word translation, word alignment, and beam-search challenges.The survey notes that increasing training data can improve deep-learning results, while uncommon words are translated poorly.Source: §3 · pp. 6–7
  • Motion generation needs more diverse data and faces graph scalability and computational costs; GAN training also faces mode collapse, non-convergence, instability, and hybrid-model complexity.The survey suggests GCNs and refining algorithms as possible ways to adjust motion-graph structure automatically.Source: §3 · pp. 7–8

Limitations

  • Multi-signer generation and high-resolution, photorealistic continuous signing remain unresolved; most reviewed models produce low-resolution samples.Avatar approaches can generate high-resolution samples but are complex and expensive; motion-graph pruning also needs to account for motion duration and speed.Source: §3 · p. 8

Future directions

  • The survey points to self-attention, multimodal fusion, structured spatio-temporal models, and sign-language prior knowledge, while calling for fast processing, standardization, and interoperability for real-world use.It also highlights assistive technologies across hearing, alerting, and communication-support devices.Source: §3 · p. 8

Related papers

Loading 2103.15910v1…