Source-linked AI summary
Machine Translation for Sign Languages
Ozge Mercanoglu Sincan, Anton Pelykh, Edward Fish, Harry Walsh, JianHe Low, Karahan Sahin, Oline Ranum, Sobhan Asasi, Steven Emery, Richard Bowden
TL;DR
Sign-language MT faces scarce data, inadequate evaluation, and the challenge of representing simultaneous, spatial linguistic structure. This review synthesizes technical, linguistic, and ethical perspectives, concluding that progress requires sustained collaboration with deaf communities and across disciplines.
Problem
Sign-language MT faces scarce datasets, limited modeling of linguistic properties, and evaluation measures that do not establish whether output is understood or accepted by deaf users.
Source: §1 · p. 2, §6 · p. 13Method
The manuscript reviews sign-language MT through technical, linguistic, and social perspectives, covering linguistic foundations, ethics, systems, and ongoing challenges.
Source: §1 · p. 3Results
The review concludes that sign-language MT requires sustained collaboration across computer vision, linguistics, and deaf communities rather than benchmark optimization alone.
Source: §6 · p. 14Takeaways & Limitations
Modeling should draw on linguistic insight, evaluation should include deaf signers, and communities should shape data governance from the outset.
Source: §6 · p. 14BLEU and similar n-gram metrics penalize valid paraphrases and alternative word orders, and can miss dense semantic content in morphologically rich languages.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Sign language machine translation has progressed substantially over the past decade, evolving from isolated sign recognition to end-to-end translation systems. Advances in pose estimation, transformer architectures, and large-scale dataset collection have driven progress, yet challenges remain. Datasets are limited compared to spoken-language resources; evaluation metrics inadequately capture the linguistic quality of output; and models must capture the simultaneous, multi-layered, and three-dimensional structure of sign languages. This manuscript provides a comprehensive review that seeks to balance technical challenges with stakeholder considerations. We examine the linguistic properties that make sign languages computationally unique, trace the evolution of recognition, translation, and production systems, and analyze ongoing technical challenges. Crucially, we address ethical considerations around data governance, community involvement, and appropriate use. Drawing on interdisciplinary perspectives spanning computer vision, sign language linguistics, and deaf studies, our analysis emphasizes that continued progress requires sustained collaboration across these fields and with deaf communities.
1. INTRODUCTION
Sign-language MT is challenging because sign languages express meaning through simultaneous visual-spatial channels. The field has moved from isolated-sign recognition toward end-to-end translation, while data, evaluation, and stakeholder challenges remain.
1. INTRODUCTION
- Sign languages coordinate hands, face, and body, use three-dimensional space, and express multiple linguistic layers simultaneously, creating distinct computational challenges.Their visual-spatial structure places sign-language MT at the intersection of linguistics, computer vision, NLP, and human-computer interaction.Source: §1 · p. 2
- Systems evolved from glove- and motion-capture-based isolated-sign recognition to video recognition and end-to-end sign-to-text and text-to-sign pipelines.Deep learning improved visual feature extraction and temporal modeling; end-to-end sign-to-text was a milestone (Camgoz et al. 2018).Source: §1 · p. 2
- Performance remains limited: datasets are scarce, expensive to annotate, and often drawn from broadcast interpretations, while key linguistic properties remain only partially modeled or evaluated.Reported gains can be tightly coupled to datasets, domains, and recording conditions (Sincan et al. 2025).Source: §1 · p. 2
- This review integrates technical, linguistic, and social perspectives, covering linguistic foundations, ethics, technology development, technical challenges, and future directions.It includes newer developments such as gloss-free translation, LLM-based pipelines, and diffusion-based production.Source: §1 · p. 3
2. LINGUISTIC FOUNDATIONS OF SIGN LANGUAGES
Sign languages convey information through coordinated manual and non-manual channels, with linguistic meaning expressed in parallel and through space. Their productive, context-dependent structures challenge systems built around fixed vocabularies and linear sequences.
Multimodality and Simultaneity
- Hands, face, and torso jointly convey linguistic information, often in parallel; models therefore need to synchronize visual cues across channels.A single parameter change can alter lexical or grammatical meaning, and eyebrow raises or head shakes can mark clause type or negation.Source: §2 · p. 3
Sign Classes
- Non-lexicalized signs such as pointing and depicting signs create an open-vocabulary problem that fixed gloss dictionaries cannot adequately represent.Lexical signs may account for less than 60% of discourse (Fenlon et al. 2014).Source: §2 · pp. 3–4
Spatial Grammar and Discourse Structure
- Interpreting signing space requires context: a motion trajectory may be part of a lexical sign or serve a grammatical function such as agreement or spatial depiction.Referents can be established in signing space and later recalled through pointing, gaze, or movement; classifier constructions combine handshape and movement to convey meaning.Source: §2 · p. 4
- Sign phrases have relatively flexible ordering, and grammatical functions such as adverbial modification may be expressed by changing a verb’s manual parameters rather than adding a separate sign.Classifier constructions and role shift further rely on coordinated movement, handshape, posture, gaze, and facial expression.Source: §2 · p. 4
3. ETHICS and SAFETY
The review frames sign-language AI as a sociotechnical issue involving language rights, data governance, and community participation. It emphasizes diverse deaf involvement in research and deployment, alongside safeguards for consent, linguistic integrity, and appropriate use.
“Nothing About Us Without Us”
- Deaf people should hold real decision-making roles from project outset, with governance reflecting the diversity of deaf communities and not only prominent representatives.Suggested roles include investigators, advisory-board members, and paid consultants; representation should include Black and Global Majority deaf people, deafblind signers, LGBTQIA+ signers, and people from different regions and backgrounds.Source: §3 · pp. 4–5
Data Governance and Consent
- Sign-language video is biometric and cultural data, so responsible collection calls for accessible sign-language consent, bounded reuse, NIL-rights policies, and fair remuneration.NIL policies address whether and under what conditions a person’s face, body, and signing style may be used to create a photorealistic avatar.Source: §3 · p. 5
Linguistic Integrity and Diversity
- Biased data can erase regional and cultural variation, while simplified or spoken-language-influenced output can mislead non-signers and undermine sign-language status.Large-scale resources often derive from broadcast interpretations rather than native deaf signing (Fox et al. 2023; De Meulder 2025).Source: §3 · p. 5
Appropriate Use
- AI should support access when interpreters are unavailable, not replace qualified interpreters in high-stakes settings; low-risk uses require deaf-expert quality verification.The text names healthcare, legal interviews, and immigration hearings as high-stakes contexts.Source: §3 · p. 5
Governance and Accountability
- Responsible deployment requires transparent documentation of training data, limitations, and intended uses, plus channels for complaints, redress, or opting out.Deployment should also be monitored for effects on interpreters and deaf professionals, including possible budget cuts to interpreting services.Source: §3 · p. 6
Summary
- Careless decisions risk entrenching audism and unequal access, while careful use may expand access and choice.The review situates sign-language AI at the intersection of disability rights, language rights, and digital justice.Source: §3 · p. 6
4. HISTORICAL DEVELOPMENT OF SIGN LANGUAGE TECHNOLOGY
The section traces sign language technology from early sensing and rule-based systems through deep-learning recognition, translation, and production. Across these areas, methods and datasets have shifted toward larger-scale visual learning, while gloss dependence and representational challenges remain.
4.1. Datasets
- Datasets shifted from intrusive wearable sensors to monocular RGB, while later collections expanded vocabulary, signer diversity, and in-the-wild coverage.Continuous corpora range from gloss-aligned benchmarks to sentence-aligned, gloss-free collections.Source: §4.1 · pp. 6–7
4.2. Sign Language Recognition
- Recognition evolved from handcrafted classifiers to neural, transformer, and pose-based models, with pretraining and weaker supervision addressing limited labels and continuous sequences.Continuous recognition commonly predicts gloss sequences without explicit temporal boundaries; recent work uses frame-level pseudo-labels.Source: §4.2 · pp. 8–9
4.3. Sign Language Translation
- Translation moved from gloss-mediated pipelines toward gloss-free visual-to-text models, with contrastive pretraining, language models, and scaling among recent directions.Gloss pipelines depend on costly annotations and omit some signing information, while gloss-free approaches directly map visual features to text.Source: §4.3 · pp. 9–10
4.4. Sign Language Production
- Production approaches moved from modular text-to-gloss and gloss-to-pose systems toward neural generation, motion-token representations, and diffusion-based methods.Gloss intermediaries can bottleneck information; direct text-to-pose approaches may under-articulate motion, while glosses can remain auxiliary supervision.Source: §4.4 · pp. 10–11
5. TECHNICAL CHALLENGES
Technical challenges span limited and costly data, imperfect evaluation, restricted generalization, computational demands, and linguistic structures that current systems model incompletely.
Data Availability and Annotation
- Sign-language datasets remain far smaller than spoken-language resources, costly to collect and annotate, and often based on interpreted content with approximate sentence alignment.These conditions complicate training, evaluation, and representativeness of natural signing.Source: §5 · pp. 11–12
Evaluation Methodology
- Borrowed text-translation metrics can penalize valid paraphrases and miss error severity, while sign production lacks an agreed metric for linguistic quality, comprehensibility, or naturalness.Human evaluation is costly and difficult to standardize; back-translation may reward machine-readable but unnatural signing and cannot assess non-manual features.Source: §5 · p. 12
Real-World Generalization
- Controlled benchmarks poorly represent signer, style, and conversational variation, while visual clutter and pose-estimation limits can further hinder real-world generalization.Skeleton and mesh methods may decouple motion from appearance, but estimation error and extraction cost constrain their use.Source: §5 · p. 12
Computational Constraints
- Current state-of-the-art systems do not provide high-quality real-time translation, and preprocessing, LLM inference, and mobile-device constraints add costs.Distillation, quantization, and efficient architecture design offer partial solutions, but acceptable quality remains difficult under these constraints.Source: §5 · p. 13
Linguistic Complexity
- Most state-of-the-art approaches omit non-manual features as an explicit modality, while fixed vocabularies and template matching challenge productive morphology.Spatial reference and coreference require maintaining discourse referents across context, but current methods are constrained to short context windows by multimodal processing and memory costs.Source: §5 · p. 13
6. CONCLUSION
The review surveys technical advances alongside persistent gaps in data, evaluation, and linguistic modeling. It concludes that progress requires collaboration with deaf communities as well as across technical and linguistic disciplines.
6. CONCLUSION
- Recent systems use pose-based representations, transformers, self-supervised learning, and large language models, while neural rendering and diffusion improve sign production.Large-scale broadcast and web datasets have also become more available.Source: §6 · p. 13
- Benchmark performance can remain disconnected from practical usefulness because available data often represents broadcast interpretation and common metrics do not measure deaf users’ understanding or acceptance.The review identifies simultaneity, spatial grammar, and non-manual marking as linguistic properties that remain poorly modeled or ignored.Source: §6 · p. 13
- The authors argue that linguistic insight, deaf signers’ evaluation, and community-shaped data governance require sustained collaboration across computer vision, linguistics, and deaf communities.They describe community involvement as both ethical grounding and a methodological correction for identifying relevant problems and success criteria.Source: §6 · p. 14
DISCLOSURE STATEMENT
The authors report no known affiliations, memberships, funding, or financial holdings that could be perceived as affecting this review’s objectivity.
DISCLOSURE STATEMENT
- The authors report no known affiliations, memberships, funding, or financial holdings that might affect the review’s objectivity.Source: DISCLOSURE STATEMENT · p. 14