Source-linked AI summary
Progressive Transformers for End-to-End Sign Language Production
Ben Saunders, Necati Cihan Camgoz, Richard Bowden
TL;DR
Continuous sign language production requires translating spoken sentences into full sign sequences rather than concatenated isolated signs. The paper proposes Progressive Transformers with counter decoding, two model configurations, augmentation, and back-translation evaluation; on PHOENIX14T, direct text-to-pose translation can outperform gloss-mediated production.
Problem
Previous SLP work focused on concatenated isolated signs, while continuous-output translation remains relatively underresearched.
Method
Progressive Transformers translate spoken text into continuous 3D sign poses using counter decoding, with direct and gloss-mediated configurations.
Results
Direct text-to-pose translation can outperform a gloss intermediary model, while augmentation techniques reduce model drift and improve SLP performance.
Takeaways & Limitations
Direct text-to-pose SLP is not limited to training data with expensive gloss annotations.
Takeaways & Limitations
Future work must expand production to non-manual aspects such as body pose, facial expressions, and mouthings.
Abstract
from arXiv · showhide
The goal of automatic Sign Language Production (SLP) is to translate spoken language to a continuous stream of sign language video at a level comparable to a human translator. If this was achievable, then it would revolutionise Deaf hearing communications. Previous work on predominantly isolated SLP has shown the need for architectures that are better suited to the continuous domain of full sign sequences. In this paper, we propose Progressive Transformers, a novel architecture that can translate from discrete spoken language sentences to continuous 3D skeleton pose outputs representing sign language. We present two model configurations, an end-to-end network that produces sign direct from text and a stacked network that utilises a gloss intermediary. Our transformer network architecture introduces a counter that enables continuous sequence generation at training and inference. We also provide several data augmentation processes to overcome the problem of drift and improve the performance of SLP models. We propose a back translation evaluation mechanism for SLP, presenting benchmark quantitative results on the challenging RWTH-PHOENIX-Weather-2014T(PHOENIX14T) dataset and setting baselines for future research.
1 Introduction
The paper addresses continuous sign language production by introducing Progressive Transformers for translating spoken text into 3D sign pose sequences. It compares gloss-mediated and direct text-to-pose configurations and evaluates them with back translation on PHOENIX14T.
- Previous SLP work mainly concatenated isolated signs, motivating architectures for full continuous sign sequences.
- Progressive Transformers translate spoken language directly into continuous 3D sign pose sequences in an end-to-end manner.
- Counter decoding tracks production progress to generate variable-length continuous sequences without a predefined vocabulary or explicit end-of-sequence token.
- The models use either a gloss intermediary in T2G2P or direct spoken-language-to-pose translation in T2P.
- The study proposes back translation evaluation and benchmark results on the challenging PHOENIX14T dataset, alongside qualitative results for unseen text.
2 Related Work
Prior SLP systems relied on avatars, rules, isolated-sign generation, or fixed-length outputs, while continuous-output translation remains underresearched. Progressive Transformers target this gap with dynamic continuous sequence generation.
- SLT differs from CSLR because sign and spoken languages have different grammars and word order, making translation more challenging.
- Earlier SLP systems used phrase lookup, pre-generated avatar sequences, or static rule-based processing.
- A prior deep-learning system generated concatenated isolated 2D skeleton poses from glosses through independently trained processes and a lookup table.
- Unlike a fixed seven-frame sign per source word, the proposed model learns dynamic output length and word ordering from data.
- Neural Machine Translation:: Continuous-output translation remains relatively underresearched, with prior methods using fixed sizes or binary or continuous EOS representations.
3 Progressive Transformers
Progressive Transformers translate spoken-language text or gloss into continuous 3D sign pose sequences. The architecture combines symbolic or direct text translation with progressive decoding and counter-based sequence-length prediction.
- 3 Progressive Transformers: The paper addresses variable-length, non-monotonic translation from discrete spoken language into continuous 3D sign pose sequences.Sign outputs contain many more frames than source words and occupy a continuous vector space.
- 3 Progressive Transformers: Progressive Transformers support both a gloss-mediated Text to Gloss to Pose model and an end-to-end Text to Pose model.The symbolic transformer maps text to gloss, while the progressive transformer maps symbolic input to continuous pose output.
- 3.1 Symbolic Transformer: The symbolic transformer uses encoder-decoder attention to align source text with target gloss sequences and generate target predictions autoregressively.Source and target tokens receive linear embeddings and positional encoding before processing.
- 3.2 Progressive Transformer: The progressive transformer embeds each 3D pose frame and concatenates its joint embedding with a counter embedding representing temporal position.The counter value ranges from 0 to 1 relative to the total sequence length.
- 3.2 Progressive Transformer: Counter decoding predicts one pose frame and its counter value per step, ending generation when the predicted counter reaches 1.The completed predicted sequence is trained against the ground-truth sequence using mean squared error.
4 Quantitative Experiments
The experiments use PHOENIX14T sign videos converted into normalized 3D skeleton poses and evaluate production through back translation to spoken language.
- Dataset: PHOENIX14T provides parallel sign videos and German translations from 9 signers, comprising 8,257 videos, 2,887 German words, 1,066 glosses, and 835,356 frames.
- Pose representation: 2D joints are extracted with OpenPose, lifted to 3D, refined with inverse kinematics, normalized, and represented by x, y, and z coordinates.
- Evaluation: Back translation uses a pose-input SLT model to generate spoken-language translations, scored with BLEU and ROUGE, including BLEU n-grams from 1 to 4.
- Model setup: The experimental setup evaluates symbolic and progressive transformers using two layers, eight heads, embedding size 256, Xavier initialization, Adam, and learning rate 10^-3.
4.3 Symbolic Transformer: Text to Gloss
The symbolic transformer experiment evaluates text-to-gloss translation and reports state-of-the-art performance against the Stoll et al. recurrent baseline.
- Results: The symbolic transformer achieves state-of-the-art text-to-gloss results and significantly outperforms Stoll et al.’s encoder-decoder network.
- Baseline: Stoll et al.’s comparison system uses an encoder-decoder network with four layers of 1000 GRUs.
- Evaluation: Table 1 reports the symbolic transformer results for Text to Gloss translation.
4.4 Progressive Transformer: Gloss to Pose
The progressive transformer experiments address prediction drift in continuous gloss-to-pose generation through future prediction, counter-only inputs, and noise augmentation.
- Experimental comparison: The experiments compare progressive gloss-to-pose production against an unaugmented base model and report results in Table 2.
- Drift: Prediction drift arises when inference errors accumulate because the model conditions on its own previous continuous pose predictions.
- Future Prediction: Predicting all next 10 frames improves performance over the unaugmented base architecture by requiring broader movement structure rather than only the previous frame.
- Just Counter: The pure memorisation setup supplies only counter values instead of full 3D joint positions and considerably increases BLEU-4.
- Just Counter: Using only counter values makes training and inference inputs identical, reducing reliance on ground-truth joint embeddings.
Gaussian Noise
Combining future prediction with Gaussian noise produces the best reported BLEU-4 performance and is retained for subsequent experiments.
- Results: The combination of future prediction and Gaussian noise achieves the best BLEU-4 performance in Table 2.
- Interpretation: This combined setup trains the model to handle both multi-frame prediction and noisy inputs, building robustness to drift.
4.5 Text2Pose v Text2Gloss2Pose
The direct Text-to-Pose configuration outperforms the gloss-mediated alternative, which may lose contextual information from the source text. Its success also indicates that sign production can be trained without gloss annotations.
- Text to Pose (T2P) outperforms Text to Gloss to Pose (T2G2P).
- Gloss mediation can bottleneck production by omitting contextual information available in spoken language.
- The successful T2P network maps spoken language to sign representations and produces accurate sign pose sequences.
- The direct configuration is useful where sign-language domains lack expensive gloss annotations.
- The final score is 9.94 BLEU-4, exceeding similar end-to-end Sign to Text methods without gloss information.The paper notes that this is an unfair direct comparison but considers it an indication of model performance and pose quality.
5 Qualitative Experiments
Qualitative examples show that the model produces visually realistic, smooth sign pose sequences that closely correspond to ground-truth video. Proper nouns and specific entities remain especially difficult.
- The model produces visually pleasing, realistic-looking sign with close correspondence to the ground-truth video.
- Body motion is smooth and accurate, while hand shapes are meaningful but somewhat under-expressed.
- Proper nouns and specific entities are the most difficult productions because training data provide limited grammatical context and examples.
- Neighbouring predicted joint locations are closely positioned, indicating learned subtle signer movement and smooth transitions between signs.
6 Conclusion
The paper concludes that Progressive Transformers generate continuous sign pose sequences from spoken language through counter decoding and two network configurations. Experiments support direct text-to-pose production without requiring gloss annotations, while future work targets richer non-manual signing.
- Progressive Transformers translate discrete spoken language into continuous sign pose sequences using counter decoding without an explicit end-of-sequence token.
- The paper presents direct text-to-pose and stacked text-to-gloss-to-pose configurations.
- Experiments on PHOENIX14T use back translation for evaluation and report data augmentation as reducing model drift and improving SLP performance.
- Direct text-to-pose can outperform gloss-mediated production, allowing training without expensive gloss annotations.
- Future work will expand production to non-manual aspects including body pose, facial expressions, and mouthings.
Progressive Transformers for End-to-End Sign Language Production: Supplementary Material
The supplementary material provides additional qualitative examples comparing spoken-language inputs, produced sign poses, ground-truth poses, and original videos. It includes examples of German weather-forecast sentences.
- The supplementary material adds qualitative Progressive Transformer SLP examples in Figures 1 and 2.
- Each example compares spoken-language input, produced sign pose, ground-truth pose, and original video.
- The examples include German weather-forecast sentences with English translations.