Source-linked AI summary
A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, Stephen Lin
TL;DR
Sign language translation is limited by scarce parallel data, and gloss-based pipelines can omit visual information such as facial expressions. The paper progressively pretrains visual and language modules from general to within-domain data, then connects them with a visual-language mapper; this simple approach outperforms existing methods on two benchmarks.
Problem
Existing sign language datasets contain fewer than 20K parallel samples, far below the roughly 1M samples typically needed for effective neural machine translation.
Method
The method progressively pretrains Sign2Gloss and Gloss2Text modules from general domains to within-domain tasks, then jointly fine-tunes them through a visual-language mapper.
Results
The simple framework surpasses all existing methods on the PHOENIX-2014T and CSL-Daily sign language translation benchmarks.
Takeaways & Limitations
Progressive transfer learning from human action and spoken-language resources provides a simple, effective baseline for sign language translation.
Takeaways & Limitations
The two-stage gloss-based pipeline can omit spatio-temporal visual information, including facial expressions that convey linguistic information.
Abstract
from arXiv · showhide
This paper proposes a simple transfer learning baseline for sign language translation. Existing sign language datasets (e.g. PHOENIX-2014T, CSL-Daily) contain only about 10K-20K pairs of sign videos, gloss annotations and texts, which are an order of magnitude smaller than typical parallel data for training spoken language translation models. Data is thus a bottleneck for training effective sign language translation models. To mitigate this problem, we propose to progressively pretrain the model from general-domain datasets that include a large amount of external supervision to within-domain datasets. Concretely, we pretrain the sign-to-gloss visual network on the general domain of human actions and the within-domain of a sign-to-gloss dataset, and pretrain the gloss-to-text translation network on the general domain of a multilingual corpus and the within-domain of a gloss-to-text corpus. The joint model is fine-tuned with an additional module named the visual-language mapper that connects the two networks. This simple baseline surpasses the previous state-of-the-art results on two sign language translation benchmarks, demonstrating the effectiveness of transfer learning. With its simplicity and strong performance, this approach can serve as a solid baseline for future research. Code and models are available at: https://github.com/FangyunWei/SLRT.
1. Introduction
Sign language translation is constrained by scarce parallel data and the loss of visual information in discrete glosses. The paper addresses these limitations through progressive pretraining and a visual-language mapper that jointly connects visual and language representations.
- Data bottleneck: Existing sign language datasets contain fewer than 20K parallel samples, compared with roughly 1M samples typically needed for effective neural machine translation.The paper identifies training-data scale as a key factor limiting sign language translation progress.
- Problem formulation: Sign language translation separates into Sign2Gloss visual recognition and Gloss2Text language translation, enabling each task to be pretrained independently.Sign2Gloss maps videos to semantic glosses, while Gloss2Text maps glosses to spoken-language text.
- Approach: The method progressively pretrains visual and language modules from general domains to within-domain tasks before fine-tuning the joint model.Visual pretraining uses human action recognition followed by sign-language data, while language pretraining uses multilingual and gloss-to-text data.
- Information loss: A two-stage Sign2Gloss2Text pipeline can lose facial-expression information because glosses are discrete representations that omit some spatio-temporal visual cues.The paper notes that facial expressions can convey linguistic information, such as the adverb “Extremely.”
- Approach: The visual-language mapper connects visual features before gloss classification to gloss embeddings in the translation model, allowing joint optimization without discrete gloss representations.It is implemented as a fully connected MLP with two hidden layers.
- Results: The simple transfer-learning framework surpasses existing methods on both PHOENIX-2014T and CSL-Daily, including methods using semi-supervised learning.The comparison is reported without handcrafted multi-cue integration or advanced augmentation such as back-translation.
2. Related Work
Prior work draws on sign recognition, action recognition, and pretrained language models, but this paper combines general-domain and within-domain supervision through progressive pretraining. It also emphasizes that pretrained language models had not previously shown demonstrated effectiveness in this setting.
- Sign Language Recognition: Sign language recognition includes isolated recognition of one gloss and continuous recognition of gloss sequences from continuous videos with weak sentence-level annotations.The paper transfers within-domain knowledge from isolated sign recognition to continuous recognition and translation.
- Action Recognition: Action-recognition research supplies visual modeling resources because fine-grained gesture understanding is a special case of human action recognition.The paper combines action-recognition pretraining with within-domain Sign2Gloss pretraining.
- Pretraining for Text Generation: Large-scale monolingual language-model pretraining has produced substantial gains in low-resource neural machine translation.The paper uses multilingual denoising pretraining through mBART for its translation network.
- Pretraining for Text Generation: Compared with prior work using pretrained word embeddings, the paper leverages pretrained language models and reports significant improvement.The related-work discussion states that earlier word-embedding approaches failed to demonstrate effectiveness.
3. Method
The method progressively pretrains separate visual and translation networks, then jointly fine-tunes them through a visual-language mapper. This design transfers knowledge from action-recognition and multilingual translation domains while preserving visual information beyond discrete glosses.
- Progressive pretraining: The framework decomposes sign language translation into Sign2Gloss and Gloss2Text, enabling separate pretraining before joint fine-tuning.Sign2Gloss converts sign videos into semantic glosses, while Gloss2Text maps glosses to spoken-language text.
- Motivation and outcome: The simple framework achieves high sign language translation performance and supports leveraging training data from both vision and language pretraining domains.The authors identify limited training data as a central bottleneck in current sign language translation systems.
- Visual encoder: The visual encoder transforms raw video into dense features for gloss prediction using an S3D backbone and lightweight temporal head.The backbone produces temporally dense features, which the head further encodes for gloss sequence prediction.
- Visual encoder: Visual pretraining progresses from generic human actions on Kinetics-400 to isolated sign recognition on WLASL, followed by Sign2Gloss training on continuous gloss annotations.This sequence transfers general action knowledge into fine-grained, within-domain gloss prediction.
- Translation network: The translation network uses mBART initialization and further within-domain Gloss2Text pretraining to map gloss sequences to spoken-language sentences.The translation model is a sequence-to-sequence Transformer initialized from multilingual denoising pretraining.
- Joint training: The visual-language mapper is a two-hidden-layer MLP that converts visual features into language features for the translation encoder during joint training.It connects the visual and language networks while avoiding reliance on the discrete gloss representation during joint optimization.
4. Experiments
Experiments evaluate the method on PHOENIX-2014T and CSL-Daily, then isolate the effects of visual pretraining, language pretraining, and joint multi-modality training. The results show strong benchmark performance and benefits from progressive pretraining and visual-language integration.
- Datasets and evaluation: PHOENIX-2014T contains 8k video-gloss-text triplets, while CSL-Daily contains 20k triplets; evaluation covers Sign2Gloss, Gloss2Text, Sign2Gloss2Text, and Sign2Text.Sign2Gloss uses WER, while the translation tasks use ROUGE and BLEU.
- Comparison with state-of-the-art methods: The framework outperforms all compared methods by large margins on PHOENIX-2014T and surpasses BN-TIN-Transf + BT on CSL-Daily without extra manually generated data.The comparisons include methods using multi-cue features, back translation, and semi-supervised learning.
- Pretraining of visual encoder: K-400 pretraining reduces PHOENIX Sign2Gloss WER from 28.06 to 23.50, and K-400→WLASL further reduces it to 22.45.These settings progressively add general-domain action recognition and within-related-domain sign recognition pretraining.
- Pretraining of visual encoder: Skipping within-domain Sign2Gloss pretraining reduces Sign2Text BLEU-4 by nearly 7 on both evaluated sets.Visual pretraining gains become narrower for Sign2Text, indicating that translation-model information is also complementary.
- Pretraining of translation model: CC25 pretraining achieves 26.95 BLEU-4 on PHOENIX Sign2Text, while progressive pretraining achieves the best result.The paper attributes the best result to general-domain pretraining plus within-domain alignment through Gloss2Text and Sign2Text training.
- Joint multi-modality training: Joint Sign2Text training with gloss representations or gloss logits outperforms both the Sign2Gloss2Text pipeline and the Gloss2Text upper bound.The ablation compares gloss logits, gloss representations, and S3D features as inputs to the visual-language mapper.
5. Conclusion
The paper presents a simple multi-modality transfer learning baseline that progressively pretrains visual and language modules, then connects them with a Visual-Language Mapper for joint sign language translation training.
- The method progressively transfers external knowledge from human action and spoken language domains into visual and language modules.The modules are pretrained from general domains to within-target domains to address data scarcity.
- The individually pretrained modules are bridged through the Visual-Language Mapper for joint sign language translation training.
- Experiments on two sign language translation datasets show that the approach outperforms all state-of-the-art methods.
- The framework is applicable to various sign languages and is intended to facilitate future research.
A. Implementation Details
Training uses standard optimization settings, general-domain visual pretraining, and a Visual-Language Mapper that bridges visual features with language representations while freezing the visual backbone during joint training.
- Training uses batch size 8, Adam with 1e-3 weight decay, and cosine annealing scheduling.
- General-domain visual pretraining uses K-400 and WLASL, following the procedure of prior work.
- The Visual-Language Mapper bridges visual-modality features and language-modality representations.
- The S3D backbone is frozen during joint training to reduce computational cost.
- For PHOENIX-2014T, gloss representations provide the Visual-Language Mapper input, while CSL-Daily glosses often contain most information needed to generate text.In CSL-Daily, text can often be predicted by reordering or copying the gloss sequence.
B. More ablations
Ablations examine loss weighting and temporal downsampling, finding that translation performance is insensitive to the tested loss weights and comparing reduced video sampling rates.
- The method is insensitive to the tested relative weights of CTC and cross-entropy losses in Sign2Text joint training.
- Temporal downsampling rates of 1/2 and 1/3 are evaluated for PHOENIX-2014T Sign2Gloss and Sign2Text training.
C. Qualitative Analysis
Qualitative analyses compare end-to-end Sign2Text with gloss-based pipelines and document both its ability to use visual information and its failure cases on rare entities and long inputs.
- Effectiveness of Joint Sign2Text Training: The qualitative study compares Gloss2Text, Sign2Gloss2Text, and end-to-end Sign2Text using glosses, text references, predictions, and translations.
- Effectiveness of Joint Sign2Text Training: When Sign2Gloss predicts an incorrect gloss, the two-stage Sign2Gloss2Text pipeline can propagate that error into the translation.One example maps ‘Selfish’ to ‘Happiness’ and produces an incorrect translation.
- Effectiveness of Joint Sign2Text Training: End-to-end Sign2Text can mitigate gloss-induced errors and predict words absent from gloss annotations by using visual information.Examples include correctly translating ‘Selfish’ and adding words such as ‘dominate’ and ‘half’.
- Limitations: The method has difficulty identifying numbers and location entities because they occur infrequently in the training corpus.
- Limitations: For long inputs, translations may omit information or become less fluent.
- Limitations: The approach relies on continuous sentence-level gloss annotations, which are expensive to obtain.
- Limitations: PHOENIX-2014T and CSL-Daily use constrained recordings with limited vocabulary and signers, motivating evaluation in more diverse conversational settings.