Source-linked AI summary
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, Houqiang Li
TL;DR
SLT is limited by scarce parallel sign-text data, while collecting and annotating sign videos is expensive. The paper introduces SignBT to convert monolingual text into synthetic sign-text pairs and contributes CSL-Daily; experiments report significant improvement in SLT models.
Problem
SLT has limited parallel sign-text data, while collecting and annotating sign videos at large scale is expensive.
Method
SignBT predicts gloss sequences from monolingual text, then constructs paired sign sequences by splicing aligned pieces from a gloss-to-sign feature bank.
Results
SignBT produces significant improvement in SLT models when monolingual data are incorporated through synthetic parallel training pairs.
Takeaways & Limitations
The approach supplements scarce parallel SLT data and can be applied to encoder-decoder SLT methods without changing their network architectures.
Takeaways & Limitations
Directly generating sign video from sentences remains challenging because it involves immature techniques such as skeleton prediction, gesture generation, and temporal coherence fidelity.
Abstract
from arXiv · showhide
Despite existing pioneering works on sign language translation (SLT), there is a non-trivial obstacle, i.e., the limited quantity of parallel sign-text data. To tackle this parallel data bottleneck, we propose a sign back-translation (SignBT) approach, which incorporates massive spoken language texts into SLT training. With a text-to-gloss translation model, we first back-translate the monolingual text to its gloss sequence. Then, the paired sign sequence is generated by splicing pieces from an estimated gloss-to-sign bank at the feature level. Finally, the synthetic parallel data serves as a strong supplement for the end-to-end training of the encoder-decoder SLT framework. To promote the SLT research, we further contribute CSL-Daily, a large-scale continuous SLT dataset. It provides both spoken language translations and gloss-level annotations. The topic revolves around people's daily lives (e.g., travel, shopping, medical care), the most likely SLT application scenario. Extensive experimental results and analysis of SLT methods are reported on CSL-Daily. With the proposed sign back-translation method, we obtain a substantial improvement over previous state-of-the-art SLT methods.
1. Introduction
The paper addresses scarce parallel sign-text data by introducing SignBT, which creates synthetic training pairs from monolingual text, and contributes the CSL-Daily benchmark.
- SLT translation quality is constrained by scarce parallel data: PHOENIX-2014T has fewer than 9K Sign-German pairs, versus over 77M English-German pairs for News Translation.
- The paper proposes SignBT, a two-stage inverse SLT pipeline consisting of text-to-gloss translation and gloss-to-sign feature-sequence generation.Gloss-to-sign conversion splices sign pieces from segmented video features, using CTC-based alignment to build a sign bank.
- SignBT generates synthetic parallel data by pairing monolingual target text with source sign sequences spliced from a real feature bank.The approach uses target text from a real language corpus and sign features from real videos.
- Extensive experiments report significant improvement in SLT models from incorporating monolingual data through SignBT.
- The paper contributes CSL-Daily, a large-scale Chinese SLT benchmark with native expressions, compact annotations, and clear hand details.The dataset supports sign language recognition, translation, and generation research, with evaluations of multiple SLT baselines.
- The method targets the parallel-data shortage without changing encoder-decoder architectures and is designed to enrich SLT training with external monolingual texts.
2. Related Work
Related work covers sign language recognition, encoder-decoder sign language translation, monolingual-data methods in NMT, and publicly available sign language datasets.
- Sign Language Recognition: Sign language recognition comprises isolated recognition of trimmed videos and continuous recognition of ordered gloss sequences from continuous videos.
- Sign Language Translation: SLT differs from sign language recognition through sequence learning across differing sign-language and spoken-language word orders and vocabularies.
- Monolingual Data Exploration: NMT research introduced back-translation to generate synthetic parallel data from monolingual text without changing the encoder-decoder structure.
- Sign Language Dataset: Public video-based sign language benchmarks are summarized by whether they provide sign dictionaries, continuous videos with gloss annotations, or spoken-language translations.
3. Proposed Method
The proposed method builds a gloss-to-sign feature bank from weakly aligned sign videos, then uses sign back-translation to incorporate monolingual spoken-language text into SLT training.
- Sign Bank Generation: Gloss annotations are order-consistent with sign gestures, enabling gloss-based sign embedding pre-training and sign-bank construction.The framework uses gloss-level annotations and a CTC-trained embedding layer for temporal gloss modeling.
- Sign Bank Generation: Because continuous datasets lack gloss boundary ground truth, the method estimates most probable alignment paths between sign embeddings and gloss sequences.The search excludes blank labels and can be accelerated with the Viterbi algorithm.
- Sign Bank Generation: Estimated alignments segment video embeddings into gloss pieces stored in a lookup-indexed gloss-to-sign bank, with multiple feature pieces allowed per gloss.The bank is maintained in embedding space and supports later feature-level splicing.
- SLT Framework: The SLT encoder-decoder uses positional encoding, self-attention, feed-forward layers, and decoder attention over encoded sign features and previously predicted words.The framework is transformer-based, but the back-translation approach is not limited to this architecture.
- SLT Training with Monolingual Data: Sign back-translation predicts glosses from monolingual text, splices sampled G2S feature pieces into synthetic sign sequences, and mixes them with annotated pairs for training.The decoder-side texts in synthetic pairs always come from a real corpus.
4. The Proposed CSL-Daily Dataset
CSL-Daily is introduced as a large-scale Chinese Sign Language Translation corpus designed for practical and academic research. It covers daily-life topics and provides sign glosses, spoken-language translations, and supporting resources.
- Dataset Scope: CSL-Daily targets practical and academic sign language research with content centered on the daily lives of the deaf community.Covered topics include family life, medical care, school life, banking, shopping, and social contact.
- Data Collection: Reference-guided recording used one or two signers per text, with requirements for natural and complete expression.A senior signer’s guidance videos were recorded first, and no signer repeated the same reference text.
- Data Collection: Ten native signers from the deaf community participated in recording the corpus.Four of the signers were involved in sign language education.
- Data Collection: Videos were recorded at 1920×1080 resolution and 30 FPS, with motionless beginning and ending frames removed.
- Annotation: CSL-Daily provides sign-gloss and spoken-language translation annotations produced through cooperation between native signers and the authors.Glosses assign Chinese words with similar meanings to signs and merge glosses with similar visual expressions; translations follow reference texts and gloss annotations.
- Resources: A SignDict records each non-single sign by four sign teachers and supports future sign and gloss-free translation research.The dictionary is also intended as a reference collection for qualitative analysis of continuous sign language tasks.
5. Experiments
Experiments evaluate SLT components, training choices, synthetic-data conditions, and comparisons on CSL-Daily and PHOENIX-2014T. SignBT improves translation, with effects depending on dataset, task, warm-up, and synthetic-data usage.
- Experimental Setup: WER evaluates gloss prediction, while BLEU and ROUGE-L F1 evaluate spoken-language translation; the study considers S2T and S2G2T.S2T predicts translations directly, whereas S2G2T uses gloss as an intermediate state.
- Ablation Study: BN-TIN is selected as the sign embedding layer after outperforming an I3D alternative with less spatial detail under end-to-end training.The authors use 1D batch normalization to mitigate unstable temporal activations.
- Ablation Study: Transformer-based encoder-decoder models outperform recurrent alternatives and are used as the baseline for subsequent experiments.
- Ablation Study: Performance improves with longer CSL-Daily warm-up, but larger warm-up schedules do not improve PHOENIX-2014T and slightly decrease BLEU.The selected schedules are 50 warm-up epochs for CSL-Daily and 10 for PHOENIX-2014T.
- Ablation Study: Small amounts of synthetic data improve S2T after warm-up, whereas adding synthetic data after warm-up consistently harms S2G2T.The authors attribute this difference to sparse gloss-level noise being harder to handle than dense feature-level noise.
- Ablation Study: Synthetic-data performance improves steadily as the synthetic-data volume increases, and higher-quality synthetic data yields higher SLT scores.Training with blank inputs produces only a small gain and underperforms training with synthetic sign data.
- Comparison with State-of-the-art Methods: SignBT improves PHOENIX-2014T by 2.6 BLEU-4 points on both evaluation sets.
6. Conclusion
The paper improves SLT with monolingual data through a SignBT pipeline and contributes a large-scale, diversely annotated SLT dataset. Synthetic pairs supplement scarce parallel training data without changing encoder-decoder architectures.
- SignBT converts massive spoken-language texts into source sign sequences and uses the resulting synthetic pairs as additional SLT training data.The pipeline is designed to alleviate the shortage of parallel sign-text data.
- The approach improves translation quality with monolingual data, a topic rarely investigated in SLT.
- SignBT can be applied to encoder-decoder SLT methods without changing their network architectures.
- The paper contributes a large-scale SLT dataset with diverse topics and complete annotations.