PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks

Ben Saunders, Necati Cihan Camgoz, Richard Bowden

arXiv:2103.06982v1cs.CV

TL;DR

Prior deep-learning sign production has been limited to isolated signs, despite the need for natural, understandable signing. This paper generates continuous 3D multi-channel sign sequences directly from spoken sentences, and reports improved performance with augmentation, adversarial training, and an MDN. Its text-to-pose configuration can outperform a gloss-intermediary model, while the user study finds that productions still need improvement to be fully understandable.

  • Problem

    Previous deep-learning SLP work was limited to concatenated isolated signs focused on manual features, rather than natural, understandable sign sequences.

    Source: §1 · p. 1

  • Method

    A Progressive Transformer with counter decoding generates variable-length continuous 3D multi-channel sign sequences directly from spoken language, with augmentation, conditional adversarial training, and MDN modeling.

    Source: §1 · p. 2, §7 · p. 20

  • Results

    The authors report improved performance from data augmentation, adversarial training, and an MDN, and find that direct text-to-pose translation can outperform a gloss-intermediary model.

    Source: §7 · p. 20

  • Takeaways & Limitations

    Direct text-to-pose translation means SLP is not limited to domains where expensive gloss annotation is unavailable.

    Source: §7 · p. 20

    Proper nouns and specific entities are the most difficult productions, linked to limited grammatical context and training examples.

    Source: §6 · p. 19

Abstract

from arXiv · show

Sign languages are multi-channel visual languages, where signers use a continuous 3D space to communicate.Sign Language Production (SLP), the automatic translation from spoken to sign languages, must embody both the continuous articulation and full morphology of sign to be truly understandable by the Deaf community. Previous deep learning-based SLP works have produced only a concatenation of isolated signs focusing primarily on the manual features, leading to a robotic and non-expressive production. In this work, we propose a novel Progressive Transformer architecture, the first SLP model to translate from spoken language sentences to continuous 3D multi-channel sign pose sequences in an end-to-end manner. Our transformer network architecture introduces a counter decoding that enables variable length continuous sequence generation by tracking the production progress over time and predicting the end of sequence. We present extensive data augmentation techniques to reduce prediction drift, alongside an adversarial training regime and a Mixture Density Network (MDN) formulation to produce realistic and expressive sign pose sequences. We propose a back translation evaluation mechanism for SLP, presenting benchmark quantitative results on the challenging PHOENIX14T dataset and setting baselines for future research. We further provide a user evaluation of our SLP model, to understand the Deaf reception of our sign pose productions.

1 Introduction

SLP matters because sign languages are a preferred form of communication for Deaf communities, yet prior deep-learning systems produced isolated signs focused mainly on manual features. This paper introduces end-to-end production of continuous 3D multi-channel sign sequences from spoken language.

1 Introduction

  • Earlier deep-learning SLP systems produced concatenated isolated signs focused on manual features, using fragmented Text-to-Gloss and Gloss-to-Pose stages that can lose context and omit depth and non-manual information.The cited prior systems include Stoll et al. (2020) and Zelinka et al. (2020).Source: §1 · pp. 1–2
  • The proposed model translates spoken sentences directly into continuous 3D multi-channel sign sequences, using counter decoding to track production and predict when variable-length output ends.The productions contain manual and non-manual features.Source: §1 · p. 2
  • Data augmentation is proposed to reduce prediction drift, while a spoken-language-conditioned discriminator and an MDN are used to encourage realistic, expressive production and model multimodality.The MDN uses progressive-transformer outputs to parameterize a Gaussian mixture model.Source: §1 · p. 2
  • Evaluation includes back translation through an SLT backend on PHOENIX14T, a Deaf user study, and qualitative assessment of productions for unseen text inputs.The back-translation protocol translates produced signs back into spoken language.Source: §1 · pp. 2–3
  • This extended manuscript adds an MDN formulation for expressive sign production beyond the preliminary versions presented in Saunders et al. (2020a); (2020b).Source: §1 · p. 2

2 Related Work

Prior sign-language production approaches face limits in natural, continuous output, while related recognition, translation, and sequence-generation work offers context for this paper’s approach. The authors position their work around continuous variable-length production, conditional adversarial training, and multimodal pose generation.

2.1 Sign Language Recognition & Translation

  • SLR research expanded from isolated recognition to continuous recognition, while SLT translates sign videos into spoken-language sentences and often uses neural machine translation.The reviewed SLT approaches translate directly or through a gloss intermediary.Source: §2.1 · p. 3

2.2 Sign Language Production

  • Prior SLP approaches include costly pre-recorded avatar phrases, expensive-to-scale motion capture, and rule-based statistical translation; deep-learning systems also produced robotic concatenations of isolated signs.The paper contrasts its direct text-to-pose mapping with Stoll et al.’s separately trained processes and gloss lookup.Source: §2.2 · p. 3
  • Earlier deep-learning SLP systems produced isolated or fixed-length signs, while this work targets dynamic-length sequences and synchronised mouthings and facial movements.The related work contrasts gloss-mapped 2D skeleton concatenations with learning text-to-pose mappings directly, and contrasts fixed seven-frame-per-word output with variable-length production.Source: §2.2 · pp. 3–4

2.3 Neural Machine Translation:

  • Transformer-based models are described as state of the art in SLT; this paper proposes counter decoding to track progress and implicitly learn continuous output length.The related work notes that continuous outputs require alternatives to fixed size, binary EOS flags, or continuous EOS tokens.Source: §2.1 · p. 3, §2.3 · p. 4

2.4 Adversarial Training

  • Conditional adversarial training uses a discriminator to critique productions, and this work conditions that discriminator on spoken-language input to encourage expressive sign poses.The paper situates this approach alongside conditional GANs and prior adversarial human-pose generation.Source: §2.4 · p. 4

2.5 Mixture Density Networks

  • The paper presents its transformer–MDN combination as novel and uses MDNs to model natural sign variability through multiple distributions.MDNs model conditional distributions by predicting mixture-component weights and statistics; prior applications include continuous sequence generation and human-pose tasks.Source: §2.5 · pp. 4–5

3 Continuous 3D Sign Language Production

The section frames spoken-to-sign production as a mapping from sentences to continuous 3D poses with manual and non-manual channels. It proposes a progressive transformer with counter decoding and drift-reducing augmentation, alongside conditional adversarial training and an MDN formulation.

3 Continuous 3D Sign Language Production

  • The task is to produce continuous 3D sign poses, including manual and non-manual features, from spoken-language sentences despite differences in grammar, syntax, and representation.The authors identify non-monotonic ordering, continuous target vectors without classic EOS tokens, and concurrent sign channels as central challenges.Source: §3 · p. 5
  • A Progressive Transformer maps symbolic input to continuous sign sequences, while counter decoding tracks generation progress and implicitly learns sequence length.The section also proposes data augmentation to reduce prediction drift.Source: §3 · p. 5
  • A conditional adversarial discriminator supplements the generator, and an MDN formulation is proposed to model multimodal distributions in sign production.These components are introduced to support expressive sign and multimodal modelling, respectively.Source: §3 · p. 6

3.1 Progressive Transformer

The Progressive Transformer encodes spoken-language input and autoregressively produces continuous 3D pose frames with progress counters. Its counter decoding determines when variable-length generation ends, and the encoder-decoder architecture maps symbolic input to continuous output.

3.1 Progressive Transformer

  • The Progressive Transformer adapts the classic transformer to continuous pose output and variable-length sequence prediction, tracking production progress over time.It translates from gloss or spoken-language input to continuous 3D sign-pose sequences in an autoregressive encoder-decoder architecture.Source: §3.1 · p. 6

3.1.1 Source Embeddings

  • Source tokens are linearly embedded and given sinusoidal positional encodings to represent token content and temporal position.The embedded source sequence is then used by the encoder.Source: §3.1.1 · p. 6

3.1.2 Target Embeddings

  • Each target input concatenates a 3D pose with a counter proportional to its frame’s position in the sequence; generation ends when the predicted counter reaches 1.The counter provides a way to predict sequence end without a tokenised vocabulary and carries sequence-length and sign-duration information.Source: §3.1.2 · p. 6

3.1.3 Encoder

  • The encoder stacks multi-head attention and feed-forward layers to form contextual representations of the source sequence.Attention learns token relationships across the sequence, while multiple heads produce parallel representations with different learned parameters.Source: §3.1.3 · pp. 6–7

3.1.4 Decoder

  • The autoregressive decoder predicts one continuous pose frame and its counter at each step, stopping when the counter reaches 1.It masks future frames, maps encoder representations to the continuous decoder domain, and is trained with MSE between predicted and ground-truth sequences.Source: §3.1.4 · p. 7

3.2 Data Augmentation

Autoregressive prediction can accumulate errors because training uses ground-truth inputs while inference feeds back predictions. The section proposes future prediction, counter-only inputs, and Gaussian-noise augmentation to address prediction drift.

3.2 Data Augmentation

  • Prediction drift arises as errors in autoregressive outputs propagate, while small differences between neighbouring poses can encourage copying the previous frame.The model is trained on ground-truth inputs but must handle its own predictions at inference.Source: §3.2 · pp. 7–8

3.2.1 Future Prediction

  • Future Prediction trains the model to predict multiple future frames, encouraging it to learn sign movement rather than copy the previous frame; inference uses only the next-frame prediction.The method is conditional on the preceding pose sequence.Source: §3.2.1 · p. 8

3.2.2 Just Counter

  • Just Counter supplies only counter values as target input, so pose generation cannot rely on previous-frame joint embeddings and training and inference use the same network setup.The model must produce the target pose sequence from the counter positions alone.Source: §3.2.2 · p. 8

3.2.3 Gaussian Noise

  • Gaussian Noise adds sampled joint-wise noise to training inputs, teaching the model to correct perturbed poses and adapt to erroneous inputs at inference.Joint distribution statistics are collected each epoch, and noise is sampled for the following epoch’s inputs.Source: §3.2.3 · p. 8

3.3 Adversarial Training

Regression-based production can average away articulation, so the section adds conditional adversarial training to encourage realistic sign sequences. The progressive transformer serves as generator, while a source-conditioned discriminator evaluates temporal sign realism.

3.3 Adversarial Training

  • Regression-only training can produce average hand shapes with insufficient motion; Fig. 5 illustrates how averaging valid poses yields under-articulated signing.The discriminator is introduced to address this loss of articulation.Source: Figure 5 · p. 8, §3.3 · p. 9

3.3.1 Generator

  • The generator combines regression and adversarial losses in a weighted objective, with λReg and λGAN setting their relative importance.The progressive transformer generates sign pose sequences conditioned on the source spoken-language sequence.Source: §3.3.1 · p. 9

3.3.2 Discriminator

  • The conditional discriminator combines source-language and sign-pose features, using temporal 1D convolutions to judge sequence realism rather than isolated frames.It is trained alongside the generator and produces a scalar realism score; the discriminator is discarded at inference.Source: §3.3 · p. 9, §3.3.2 · p. 10

3.4 Mixture Density Networks

Deterministic pose prediction cannot represent sign-language variation and may produce under-articulated averages. The section models multiple possible movements with an MDN, samples autoregressively from its predicted mixture, and also describes combining the MDN with adversarial training.

3.4 Mixture Density Networks

  • Deterministic prediction can collapse varied signing into under-articulated mean poses, while an MDN models multiple valid movements and their uncertainty.The paper presents this as a route to more expressive productions.Source: §3.4 · p. 10

3.4.1 Formulation

  • The progressive transformer parameterizes a Gaussian mixture, predicting component weights and pose densities conditioned on the source token.The weights are normalized with softmax, and an exponential function ensures positive variances.Source: §3.4.1 · pp. 10–11

3.4.2 Optimisation

  • Training minimizes the negative log likelihood of the ground-truth sequence under the predicted mixture distribution.The objective is defined over the U frames and M mixture components.Source: §3.4.2 · p. 11

3.4.3 Sampling

  • At inference, the model selects the highest-weight mixture component, predicts its mean pose, and feeds sampled frames autoregressively into later steps.The variance is set to zero to avoid jitter, particularly in small hand-joint positions.Source: §3.4.3 · p. 11

3.4.4 MDN + Adversarial

  • The MDN can also serve as the generator in the adversarial framework, combining negative log-posterior and adversarial generator losses.The conditional discriminator remains unchanged, and is discarded at inference.Source: §3.4.4 · p. 11

3.5 Sign Pose Sequence Outputs

The model outputs 3D sign-pose sequences that can be animated as videos or avatars. For visual comparison with references, the authors temporally align productions with DTW, which changes timing rather than content.

3.5 Sign Pose Sequence Outputs

  • The predicted 3D joints can be connected and animated as a video or avatar, with timing supplied by the progressive-transformer counter.The authors describe this animation step as trivial.Source: §3.5 · p. 11
  • DTW aligns production timing with reference sequences for visual comparison without changing production content.The model may use a different signing speed from the reference because it has its own cadence.Source: §3.5 · p. 11
  • The implementation translates a spoken sentence into sign video within seconds, but requires whole-sentence context before translation.The paper notes that this context requirement entails a delay.Source: §3.5 · p. 11

4 Experimental Setup

The experiments use PHOENIX14T and evaluate translation and pose production through back translation, alongside three task protocols from text-to-gloss through end-to-end text-to-pose.

4.1 Dataset

  • PHOENIX14T provides 8,257 videos from 9 signers, with 2,887 German words and 1,066 glosses; pose inputs are extracted in 2D, lifted to 3D, and normalized.The experiments use the original training, validation, and test split.Source: §4.1 · p. 12, Figure 8 · p. 12

4.2 Back Translation Evaluation

  • Back translation uses an SLT model to convert produced poses into spoken language, measuring recoverable translation content and serving as a relative comparison between similar configurations.The authors report close correspondence between back-translation scores and visual production quality, while noting that changing the SLT model can change absolute scores.Source: §4.2 · p. 12

4.3 Evaluation Protocols

  • The protocols test text-to-gloss translation, gloss-to-pose production, and end-to-end text-to-pose translation, with the last requiring both translation and pose generation.Text-to-pose can be evaluated without gloss annotation.Source: §4.3 · p. 13

4.4 Model Configuration

  • The progressive transformer is configured with 2 layers, 4 heads, and embedding size 512 unless otherwise stated.Training uses Xavier initialization, Adam, and an initial learning rate of 10^-3.Source: §4.4 · p. 13

5 Quantitative Evaluation

The quantitative evaluation benchmarks text-to-gloss and pose-production configurations on PHOENIX14T. It tests augmentation, adversarial training, and MDN formulations for continuous pose generation.

5.1 Text to Gloss Translation

  • The transformer significantly outperformed Stoll et al. (2020) on text-to-gloss translation, achieving state-of-the-art results.The authors present this as support for using a transformer for sign-language understanding.Source: §5.1 · p. 13

5.2.1 Data Augmentation

  • Future prediction raised BLEU-4 from 7.38 to 11.30, and predicting 5 future frames gave the best balance between forward planning and model complexity.The just-counter approach further increased BLEU-4 by removing skeleton poses from the target input and limiting feedback of erroneous predictions.Source: §5.2.1 · pp. 13–14, Table 3 · p. 14, Table 4 · p. 14
  • Gaussian noise with rn = 5 achieved 12.80 BLEU-4; larger noise rates degraded performance, while modest noise improved robustness to autoregressive prediction errors.The authors use rn = 5 in subsequent experiments.Source: Table 5 · p. 14, §5.2.1 · p. 14

5.2.2 Adversarial Training

  • Conditional adversarial training peaked at 13.13 BLEU-4 for N = 6, compared with 12.65 BLEU-4 without source conditioning; test performance was 12.76 BLEU-4.A discriminator that is too strong reduces generator performance.Source: Table 6 · p. 15, §5.2.2 · p. 15

5.2.3 Mixture Density Networks

  • 13.14 BLEU-4 validation was the MDN’s best at M = 4, narrowly above adversarial 13.13 BLEU-4, which the authors regard as similar within back-translation error.The combined MDN-adversarial model scored 12.88 BLEU-4 on development, below either method individually.Source: Table 7 · p. 15, §5.2.3 · pp. 15–16
  • Increasing MDN mixture components initially improved validation performance, but added complexity eventually outweighed the gains.At M = 4, the model reached 13.14 BLEU-4; the combined MDN-adversarial setup scored 12.88 BLEU-4 on development, though slightly better than MDN alone on test.Source: §5.2.3 · pp. 15–16, Table 7 · p. 15

5.3 Text to Pose Production

The section evaluates end-to-end text-to-pose models and compares direct production with a gloss intermediary, then reports Deaf participants’ visual and comprehension judgments.

5.3.1 Model Configurations

  • 11.54 BLEU-4 was the best Text to Pose result, achieved by the MDN model; adversarial and MDN performance was considered equivalent given the SLT system’s limits.Adding adversarial training to the MDN did not improve performance further.Source: §5.3.1 · p. 16

5.3.2 Text to Pose v Text to Gloss to Pose

  • Text to Pose outperformed Text to Gloss to Pose on development, while the gloss-intermediary model performed better on test.The authors suggest the gloss bottleneck may discard source context, while limited training samples and the smaller gloss vocabulary may affect generalization.Source: §5.3.2 · pp. 16–17
  • The authors argue that direct text-to-pose production shows the model can jointly translate and generate poses, relevant to domains without gloss annotations.The paper also reports a higher final BLEU-4 score than the cited end-to-end Sign to Text methods’ 9.94 BLEU-4, while warning that this is not a fair direct comparison.Source: §5.3.2 · p. 17

5.4.1 Visual Task

  • User feedback was mainly equal between produced and ground-truth videos, with slightly more participants preferring the productions.The evaluation compared skeletal outputs directly, but this excludes visual information such as shadow and occlusion.Source: Table 10 · p. 17, §5.4.1 · p. 17, §5.4 · p. 17
  • 15.38% of users preferred MDN productions over ground truth, compared with 8.33% for adversarial productions.Participants’ visual ratings favored MDN despite similar back-translation performance for the two configurations.Source: Table 10 · p. 17, §5.4.1 · p. 17

5.4.2 Translation Task

  • 78.57% of users chose the correct translation for MDN productions, compared with 34.72% for adversarial productions.The task presented one production and four possible spoken-language translations.Source: §5.4.2 · p. 17, Table 11 · p. 18

6 Qualitative Evaluation

The qualitative examples show smooth, realistic pose sequences with close correspondence to ground truth, while revealing under-articulation and specific failure cases.

6 Qualitative Evaluation

  • Produced signs have smooth, accurate body motion and close correspondence to ground truth, though hand shapes are somewhat under-expressed.Head movement corresponds closely to ground truth, while mouthings are slightly under-articulated.Source: §6 · pp. 18–19
  • Gaussian-noise outputs were under-expressed; adversarial training produced more expressive hand shapes, while MDN poses appeared closer to ground truth.The authors attribute the MDN difference to mixture-distribution modeling of uncertainty, which reduces mean-like productions.Source: §6 · p. 19
  • Neighboring-frame joint positions and smooth transitions between signs indicate that the model captures subtle signer motion in continuous 3D poses.The paper contrasts these transitions with discrete spoken-language generation.Source: §6 · p. 19
  • Complex handshapes, hand occlusion, and proper nouns are difficult cases; proper nouns lack grammatical context and training examples.The figure illustrates these failure types.Source: §6 · p. 19

7 Conclusions

The paper presents an end-to-end text-to-continuous-3D sign pose model and evaluates it on PHOENIX14T and with Deaf users. It reports performance gains from augmentation, adversarial training, and MDN, while user evaluation indicates that productions remain short of full understandability.

7 Conclusions

  • The Progressive Transformer uses counter decoding to generate variable-length continuous sequences by tracking production progress and predicting sequence end.Source: §7 · pp. 19–20
  • Data augmentation targets prediction drift, conditional adversarial training prompts realistic and expressive output, and MDN parameterizes a mixture Gaussian distribution from transformer outputs.Source: §7 · p. 20
  • On PHOENIX14T, data augmentation reduced model drift, adversarial training and MDN improved performance, and direct text-to-pose outperformed gloss mediation.The authors report that direct text-to-pose can avoid domains requiring expensive gloss annotation.Source: §7 · p. 20
  • User evaluation found that productions still need improvement for full Deaf comprehension, with reducing and smoothing noise identified as possible improvements.The authors attribute some noise to the data and approaches.Source: §7 · p. 20
  • Future work proposes photo-realistic signers, improved hand articulation through a hand-shape classifier, and an automatic viseme generator for mouthing patterns.Source: §7 · p. 20
Loading 2103.06982v1…