PaperlayerView exact revision on arXiv

Source-linked AI summary

Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation

Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, Richard Bowden

arXiv:2003.13830v1cs.CVcs.CLcs.HCcs.LG

TL;DR

Sign-language translation from video is challenging because sign and spoken languages differ structurally, while prior systems either relied on gloss bottlenecks or lacked direct supervision. The paper introduces Sign Language Transformers, which jointly learn recognition and translation end to end with CTC-based gloss supervision, and reports state-of-the-art results on PHOENIX14T. The approach also provides new transformer baselines for related text-to-text tasks.

  • Problem

    Sign-language translation requires modeling complex non-monotonic mappings from high-dimensional videos to spoken sentences, while prior gloss-based and direct approaches have important constraints.

  • Method

    Sign Language Transformers jointly learn p(G|V) and p(S|V) end to end, injecting intermediate gloss supervision through a CTC loss into a transformer encoder and using a translation decoder.

  • Results

    The method achieved state-of-the-art sign-language recognition and translation results on PHOENIX14T, in some cases doubling prior translation performance.

  • Takeaways & Limitations

    Joint recognition and translation improved performance across both tasks and surpassed the text-to-text translation results treated as a virtual upper bound.

  • Takeaways & Limitations

    Gloss-based translation introduces an information bottleneck because glosses incompletely annotate the original sign video and omit crucial details.

Abstract

from arXiv · show

Prior work on Sign Language Translation has shown that having a mid-level sign gloss representation (effectively recognizing the individual signs) improves the translation performance drastically. In fact, the current state-of-the-art in translation requires gloss level tokenization in order to work. We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation while being trainable in an end-to-end manner. This is achieved by using a Connectionist Temporal Classification (CTC) loss to bind the recognition and translation problems into a single unified architecture. This joint approach does not require any ground-truth timing information, simultaneously solving two co-dependant sequence-to-sequence learning problems and leads to significant performance gains. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset. We report state-of-the-art sign language recognition and translation results achieved by our Sign Language Transformers. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models, in some cases more than doubling the performance (9.58 vs. 21.80 BLEU-4 Score). We also share new baseline translation results using transformer networks for several other text-to-text sign language translation tasks.

1. Introduction

Sign language translation from video must address complex spatio-temporal and linguistic differences between sign and spoken languages. The paper introduces an end-to-end transformer architecture that jointly learns recognition and translation using gloss supervision.

  • Motivation: Sign languages combine manual and non-manual features across multiple complementary channels.These include hand shape, movement, pose, facial expression, mouth, head, shoulders, and torso movements.
  • Motivation: Video-to-spoken-language translation is difficult because sign and spoken languages differ in ordering, grammar, concurrent channels, and spatial references.The mapping is not a simple word-to-sign correspondence.
  • Challenges: Continuous sign translation requires segmentation, sign-language recognition and understanding, and spoken-language generation.These subtasks are described as currently unsolved challenges for video-based translation.
  • Prior work: Glosses improved translation by providing a mid-level tokenization layer between continuous sign videos and spoken-language generation.The prior pipeline recognized glosses before applying text-to-text neural machine translation.
  • Prior work: 18.13 vs 9.58 BLEU-4 scores: prior Sign2Gloss2Text translation substantially outperformed direct Sign2Text translation.The gloss representation reduces sequence length and helps avoid long-term dependency issues.
  • Proposed approach: Sign Language Transformers jointly learn recognition and translation with a multi-task transformer architecture trained end to end.The approach uses an encoder recognition model with CTC loss and a translation decoder operating on learned spatio-temporal representations.
  • Contributions: The paper reports state-of-the-art recognition and translation results and new transformer baselines for related text-to-text tasks.Its contributions include exploiting gloss supervision without restricting translation to spoken language.

2. Related Work

Earlier sign-language translation research was constrained by limited datasets, small text-to-text resources, and the difficulty of mapping non-monotonic sign-language structure to spoken language. Transformer advances and PHOENIX14T enabled more capable video-based translation research.

  • Earlier translation: Earlier computational-linguistic translation datasets averaged around 3000 total words and focused on text-to-text tasks.These studies used statistical machine translation methods, including work on PHOENIX12.
  • Earlier translation: End-to-end sign-language translation from video was not realized until relatively recently.Earlier conceptual systems often recognized isolated signs before constructing sentences with a language model.
  • Dataset constraints: Suitable datasets were a major obstacle because continuous sign videos lacked sufficient spoken-language annotations and often lacked human pose information.Available resources were weakly annotated or too small for broad-domain modeling.
  • Modeling challenges: Sign sentences and spoken translations are non-monotonic, so glosses and linguistic constructs do not necessarily align one-to-one with spoken-language counterparts.This made weakly annotated CSLR methods based on shared temporal order infeasible for the translation setting.
  • Dataset advances: PHOENIX14T addressed dataset limitations as the first publicly available sign-language video-to-spoken-language translation dataset.It extends the RWTH-PHOENIX-Weather-2014 continuous sign-language recognition dataset.
  • Transformer advances: Transformer networks improved translation over legacy attention-based encoder-decoder approaches while enabling fast parallelization.Their fully connected architecture made them a leading architecture for machine translation tasks.

3. Sign Language Transformers

Sign Language Transformers jointly learn gloss recognition and spoken-language translation from sign videos in one end-to-end model. The architecture uses CTC-based intermediate gloss supervision while preserving video information for autoregressive translation.

  • The model jointly learns p(G|V) and p(S|V), where glosses and spoken-language sentences are generated from sign video frames.This unified formulation addresses sequence-length mismatch and non-monotonic mappings between sign videos and spoken language.
  • Two-stage gloss pipelines introduce an information bottleneck, while direct video-to-text models lack explicit supervision for sign-language understanding.Glosses omit crucial video information, whereas direct translation has significantly lower performance on available datasets.
  • The unified model injects intermediate gloss supervision through a CTC loss in the SLRT encoder without limiting information passed to the translation decoder.The SLTT decoder uses the learned spatio-temporal representations to generate spoken-language sentences autoregressively.
  • SLRT applies self-attention and feed-forward processing to positionally encoded spatial video embeddings, producing contextual spatio-temporal representations.Residual connections and normalization follow the encoder operations.
  • CTC trains gloss recognition from weak sequence-level supervision by marginalizing over viable alignments between frame-level predictions and gloss sequences.This avoids requiring precise frame-level gloss annotations, which are rare.
  • SLTT predicts one spoken-language word at a time with masked self-attention, encoder-decoder attention, and a cross-entropy translation loss.Training minimizes a weighted sum of recognition and translation losses, controlled by λ_R and λ_T.

4. Dataset and Translation Protocols

The evaluation uses PHOENIX14T, a continuous sign-language corpus with parallel videos, glosses, and German translations. It compares direct, gloss-mediated, and joint recognition-translation protocols, including two newly introduced protocols.

  • PHOENIX14T is a large-vocabulary continuous sign-language translation corpus and a primary recent benchmark for continuous sign-language recognition.It contains parallel videos, gloss annotations, and translations.
  • The corpus includes continuous signing from 9 signers, 1066 signs, and German translations containing 2887 words.
  • Sign2Text directly translates continuous sign videos into spoken-language sentences without an intermediate gloss representation.
  • Gloss2Text translates ground-truth gloss sequences into German and serves as a virtual upper bound for available text-to-text translation technology.The paper notes that glosses are an information bottleneck because they incompletely represent multi-channel temporal signals.
  • Sign2Gloss2Text and Sign2Gloss→Gloss2Text use CSLR-generated glosses as intermediate input to text-to-text translation models.The latter reuses the best-performing Gloss2Text network trained with ground-truth glosses.
  • Sign2Gloss evaluates continuous sign-language recognition, while Sign2(Gloss+Text) jointly learns continuous recognition and translation.These are newly introduced protocols following the existing naming convention.

5. Quantitative Results

The experiments evaluate transformer-based sign language recognition and translation across text-to-text and video-based tasks. Results show gains from transformer baselines, pretrained spatial features, tuned depth, and jointly weighted recognition and translation objectives.

  • Text-to-Text Sign Language Translation: 25.35/24.54 BLEU-4 was reached on development/test sets when transformers were used for text-to-text sign language translation.The reported improvement spans all evaluated text-to-text tasks.
  • Sign2Gloss: Nearly 7% and 6% absolute WER reductions on development/test sets followed batch normalization and ReLU applied to pretrained CNN features.The remaining experiments used batch-normalized pretrained CNN features followed by ReLU.
  • Sign2Gloss: Recognition initially improved with additional transformer layers, but further depth caused over-fitting and performance degradation; three layers were used thereafter.The layer search covered one to six layers.
  • Joint Recognition and Translation: Equal recognition and translation loss weighting improved translation but degraded recognition relative to task-specific networks.The authors attribute this trade-off to scale differences between CTC and word-level cross-entropy losses.
  • Joint Recognition and Translation: Increasing the recognition loss weight improved both recognition and translation performance, demonstrating value in sharing training between the related tasks.The joint models varied the recognition and translation loss weights, with Sign2Text using zero recognition-loss weight as a baseline.
  • Comparison with Prior Methods: The Sign Language Transformers reduced test-set WER by 2% over and more than doubled prior translation results: 9.58 vs. 20.17 and 21.32 BLEU-4.The best Sign2(Gloss+Text) model also exceeded the prior Gloss2Text result, 19.26 vs. 21.32 BLEU-4.

6. Qualitative Results

Qualitative evaluation found that the generated translations generally conveyed the source information and followed standard grammar. Named entities and specific numbers were the most difficult cases.

  • Translation Quality: Generated translations generally conveyed the same information even when their exact wording differed.The qualitative examples came from the best-performing Sign2(Gloss+Text) model.
  • Translation Difficulties: Named entities such as locations and specific numbers were the most difficult translation cases.The paper links these difficulties to limited training contexts for locations and insufficient grammatical context for distinguishing numbers.
  • Translation Quality: The produced sentences followed standard grammar with surprisingly few exceptions.This assessment is reported for the qualitative translations generated from PHOENIX14T sign videos.

7. Conclusion and Future Work

Sign Language Transformers jointly learn sign language recognition and translation end to end, using gloss supervision without making translation depend on an explicit gloss bottleneck. On PHOENIX14T, the approach achieves state-of-the-art recognition and translation results, while future work targets separate modeling of faces, hands, and body.

  • Sign Language Transformers jointly learn sign language recognition and translation in an end-to-end manner.
  • CTC loss injects gloss-level supervision into the transformer encoder without creating an explicit gloss representation bottleneck.
  • The approach achieves state-of-the-art sign language recognition and translation results on PHOENIX14T.
  • Jointly learning recognition and translation improves performance across both tasks and surpasses text-to-text translation results from video representations.
  • Future work will model faces, hands, and body separately to encourage learning their linguistic relationships.
Loading 2003.13830v1…