PaperlayerView exact revision on arXiv

Source-linked AI summary

Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining

Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, Du Zhang

arXiv:2307.14768v1cs.CV

TL;DR

SLT is difficult because visual-gestural language must be translated across domains, while gloss-based methods require scarce annotations and impose an information bottleneck. GFSLT-VLP combines CLIP-style visual-language pretraining with masked self-supervision, then transfers the pretrained encoder and decoder into an end-to-end gloss-free translator. It improves BLEU-4 by at least 5 points on PHOENIX14T and 3 points on CSL-Daily over state-of-the-art gloss-free methods, with competitive PHOENIX14T performance against most gloss-based methods.

  • Problem

    SLT faces cross-domain translation difficulty, while gloss-based methods depend on scarce annotations and an information-bottlenecking intermediate gloss representation.

  • Method

    GFSLT-VLP combines CLIP-based visual-language pretraining and masked self-supervised learning, then transfers the pretrained Visual Encoder and Text Decoder into an end-to-end gloss-free architecture.

  • Results

    ≥+5 BLEU-4 on PHOENIX14T and ≥+3 on CSL-Daily versus state-of-the-art gloss-free methods, with competitive PHOENIX14T performance against most gloss-based methods.

  • Takeaways & Limitations

    The approach substantially improves gloss-free SLT while narrowing its reported performance gap with gloss-based methods.

Abstract

from arXiv · show

Sign Language Translation (SLT) is a challenging task due to its cross-domain nature, involving the translation of visual-gestural language to text. Many previous methods employ an intermediate representation, i.e., gloss sequences, to facilitate SLT, thus transforming it into a two-stage task of sign language recognition (SLR) followed by sign language translation (SLT). However, the scarcity of gloss-annotated sign language data, combined with the information bottleneck in the mid-level gloss representation, has hindered the further development of the SLT task. To address this challenge, we propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP), which improves SLT by inheriting language-oriented prior knowledge from pre-trained models, without any gloss annotation assistance. Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training (CLIP) with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual Encoder and Text Decoder from the first stage. The seamless combination of these novel designs forms a robust sign language representation and significantly improves gloss-free sign language translation. In particular, we have achieved unprecedented improvements in terms of BLEU-4 score on the PHOENIX14T dataset (>+5) and the CSL-Daily dataset (>+3) compared to state-of-the-art gloss-free SLT methods. Furthermore, our approach also achieves competitive results on the PHOENIX14T dataset when compared with most of the gloss-based methods. Our code is available at https://github.com/zhoubenjia/GFSLT-VLP.

1. Introduction

Gloss-based SLT is limited by costly gloss annotation and information loss, motivating GFSLT-VLP’s gloss-free visual-language pretraining and end-to-end translation architecture. The method reports substantially higher BLEU-4 than prior gloss-free approaches while remaining competitive with gloss-based methods.

  • Gloss-based SLT requires labor-intensive specialist annotations and introduces an information bottleneck through intermediate gloss representations.
  • VLP jointly aligns visual and textual representations while masked self-supervision trains the Text Decoder to capture sentence syntax and semantics.
  • GFSLT transfers the pretrained Visual Encoder and Text Decoder into an end-to-end architecture that directly maps visual representations to spoken sentences without intermediate supervision.
  • ≥+5 BLEU-4 on PHOENIX14T and ≥+3 on CSL-Daily over state-of-the-art gloss-free methods, without gloss annotations.
  • The paper presents VLP as the first strategy, to its knowledge, for aligning visual and textual representations in a joint semantic space for gloss-free SLT.

2. Related Works

Prior SLT work commonly uses sign glosses or recognition-derived representations, whereas gloss-free methods omit gloss supervision but face difficult cross-modal alignment. GFSLT-VLP is positioned as a way to narrow the performance gap between gloss-free and gloss-based translation.

  • Gloss-based SLT often uses continuous sign recognition to predict glosses or obtain visual representations before translation.
  • Gloss-based methods improve translation but depend on fine-grained, specialist gloss annotation and may constrain scalability through the intermediate representation.
  • Gloss-free SLT omits gloss supervision during pretraining, training, and testing, but its performance is generally lower because video and spoken-language orders differ substantially.
  • Existing gloss-free approaches include CNN-RNN modeling, attention-based feature learning, and modules for word verification, sentence generation, and cross-modal reranking.

3. Method

GFSLT-VLP combines visual-language pretraining with masked sentence reconstruction, then transfers the pretrained encoder and decoder into an end-to-end gloss-free translation model.

  • Gloss-Free Sign Language Translation: The second-stage GFSLT model initializes its parameters from the pretrained visual encoder and text decoder for direct video-to-sentence translation without gloss supervision.The transferred components are then used in an encoder-decoder-like architecture for gloss-free SLT.
  • Visual-Language Pretraining: VLP jointly pretrains visual and textual encoders to align sign-video and sentence representations in a shared multimodal semantic space.The method uses CLIP-inspired video-text pairing and similarity learning to produce language-indicated visual features.
  • Visual-Language Pretraining: The visual encoder processes frames with shared 2D CNN layers, temporal convolutional blocks, and a Transformer encoder for short- and long-term dependencies.The Transformer encoder captures long-term temporal relationships after spatial and short-term temporal processing.
  • Visual-Language Pretraining: Masked self-supervised learning trains the text decoder to restore masked words while combining complementary pretraining signals for richer representations.The approach also applies strong video augmentation, including geometric, color-space, and temporal transformations.
  • Gloss-Free Sign Language Translation: The decoder generates one word at a time from the encoder state, ending at <EOS>, while the full model is optimized with video-to-sentence cross-entropy.Sentence generation begins with <BOS> and uses Linear and Softmax layers to calculate p(S|V).

4. Experiments

Experiments evaluate GFSLT-VLP on PHOENIX14T and CSL-Daily using BLEU and ROUGE, showing strong gloss-free gains and competitive gloss-based performance. Ablations examine augmentation, fine-tuning, and training duration.

  • Comparison with State-of-the-art Methods: +5.7 BLEU-4 on PHOENIX14T Test and +3.2±0.1 BLEU-4 on CSL-Daily over gloss-free methods.On PHOENIX14T, the method also improves ROUGE by about +2.6 on Test; on CSL-Daily, ROUGE improves by about 2.2±0.2.
  • Comparison with State-of-the-art Methods: GFSLT-VLP is highly competitive with most gloss-based PHOENIX14T methods and remains close to selected gloss-based CSL-Daily systems.The PHOENIX14T comparison includes SLRT and STMC-T, while CSL-Daily comparisons include SLRT and BN-TIN-Transf without several auxiliary training or model enhancements.
  • Ablation Studies: Strong data augmentation during VLP raises PHOENIX14T Test BLEU-4 from 19.84 to 22.05, whereas lightweight augmentation yields only small gains.The reported increase is +2.2, while lightweight augmentation improves Dev/Test BLEU-4 by about +0.3/+0.1.
  • Ablation Studies: Unified fine-tuning of the visual embedding module and Transformer encoder gives a +2.23 gain, exceeding separate fine-tuning of either component.Separate fine-tuning changes are reported as -0.01 and +1.39, while decoder fine-tuning contributes at most 1 point.
  • Ablation Studies: Gloss-free training requires more than 100 epochs for satisfactory performance, while 80 pre-training epochs balance results across the two datasets.The study attributes the longer downstream training requirement to slower convergence without intermediate representations.

5. Qualitative Results

Qualitative comparisons show that both systems produce complete sentences, but VLP reduces keyword errors and improves named-entity translations. The gains include more accurate place names and months.

  • Qualitative Results: VLP more accurately translates named entities, including place names and months, than the baseline.The comparison is based on PHOENIX14T test videos.
  • Qualitative Results: The baseline is more error-prone on keywords, sometimes producing translations that differ drastically despite capturing general meaning.Both systems can understand the general meaning and produce complete sentences.

6. Conclusion and Future work

The work reduces the semantic gap between visual and textual representations through visual-language and masked self-supervised pretraining for gloss-free SLT. It finds that data scale and model parameters significantly affect performance, while identifying large-scale gloss-free pretraining as future work.

  • The proposed pretraining paradigm combines masked self-supervised learning with visual-language supervision to learn language-indicated visual representations for gloss-free SLT.
  • Experiments show that both data scale and model parameters significantly affect the method’s performance.
  • Further research is needed on pretraining with a large-scale SLT dataset without gloss annotations.

A. More Implementation details

The GFSLT model processes padded variable-length sign videos through ResNet-based visual feature extraction and temporal modules that capture short-term dependencies.

  • Variable-length sign videos are padded to the longest sequence in each batch before processing.
  • ResNet processing without a fully connected layer produces visual features with size B × T × 512.
  • Two Conv1D-BN1D-RELU-MaxPooling1D temporal modules capture short-term dependencies in the sign video.

B.1. Impact of Model Parameter Size.

For GFSLT, increasing network depth does not improve performance and instead causes more severe overfitting, likely because SLT data are limited.

  • Increasing the network depth to four layers exacerbates overfitting instead of improving results.
  • The authors attribute the deeper-network overfitting to the limited scale of SLT data.
  • A sufficiently large SLT dataset may alleviate the overfitting issue caused by adding network layers.

B.2. Impact of Mask Rate.

The masking ablation uses BERT-style token replacement and identifies a 15% masking rate as optimal for BLEU-4. The supplied passages also describe alternative pretraining behavior and parameter emphasis.

  • 15% masking achieves the optimal BLEU-4 score in the masking-rate experiment.
  • The token-masking strategy replaces 80% of selected tokens with [Mask], 10% with another token, and leaves 10% unchanged.
  • The experiments place more emphasis on pretraining the Visual Encoder than the Text Decoder.
  • Freezing the text encoder as a teacher for visual-encoder learning does not produce satisfactory pretraining results.

B.4. Impact of Loss weight

The loss-weight coefficient λ has a relatively minor effect near the tested settings, but increasing it eventually reduces performance, motivating λ=0.1. Additional ablations show that VLP and strong augmentation work best when combined across both training stages.

  • B.4. Impact of Loss weight: Performance fluctuations stayed around ±0.1 as λ varied, but increasing λ eventually caused performance to decline, so λ was set to 0.1.The study examined λ as the weight controlling the loss contribution associated with the Text Decoder.
  • B.5. Investigation VLP on CSL-Daily: On CSL-Daily, VLP improved translation performance, while adding strong augmentation only in Stage 2 without VLP reduced it.The ablation compared VLP with strong augmentation applied in Stage 1 or Stage 2.
  • B.5. Investigation VLP on CSL-Daily: The best CSL-Daily result occurred when strong data augmentation was applied in both stages together with VLP.This pattern was consistent with the experiments on Phoenix14T.
Loading 2307.14768v1…