PaperlayerView exact revision on arXiv

Source-linked AI summary

Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining

Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, Du Zhang

arXiv:2307.14768v1cs.CV

TL;DR

SLT is difficult because visual-gestural language must be translated across domains, while gloss-based methods require scarce annotations and impose an information bottleneck. GFSLT-VLP combines CLIP-style visual-language pretraining with masked self-supervision, then transfers the pretrained encoder and decoder into an end-to-end gloss-free translator. It improves BLEU-4 by at least 5 points on PHOENIX14T and 3 points on CSL-Daily over state-of-the-art gloss-free methods, with competitive PHOENIX14T performance against most gloss-based methods.

  • Problem

    SLT faces cross-domain translation difficulty, while gloss-based methods depend on scarce annotations and an information-bottlenecking intermediate gloss representation.

    Source: §1 · p. 1

  • Method

    GFSLT-VLP combines CLIP-based visual-language pretraining and masked self-supervised learning, then transfers the pretrained Visual Encoder and Text Decoder into an end-to-end gloss-free architecture.

    Source: §1 · p. 2

  • Results

    ≥+5 BLEU-4 on PHOENIX14T and ≥+3 on CSL-Daily versus state-of-the-art gloss-free methods, with competitive PHOENIX14T performance against most gloss-based methods.

    Source: §1 · p. 2, §4.3 · p. 7

  • Takeaways & Limitations

    The approach substantially improves gloss-free SLT while narrowing its reported performance gap with gloss-based methods.

    Source: §6 · p. 9

Abstract

from arXiv · show

Sign Language Translation (SLT) is a challenging task due to its cross-domain nature, involving the translation of visual-gestural language to text. Many previous methods employ an intermediate representation, i.e., gloss sequences, to facilitate SLT, thus transforming it into a two-stage task of sign language recognition (SLR) followed by sign language translation (SLT). However, the scarcity of gloss-annotated sign language data, combined with the information bottleneck in the mid-level gloss representation, has hindered the further development of the SLT task. To address this challenge, we propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP), which improves SLT by inheriting language-oriented prior knowledge from pre-trained models, without any gloss annotation assistance. Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training (CLIP) with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual Encoder and Text Decoder from the first stage. The seamless combination of these novel designs forms a robust sign language representation and significantly improves gloss-free sign language translation. In particular, we have achieved unprecedented improvements in terms of BLEU-4 score on the PHOENIX14T dataset (>+5) and the CSL-Daily dataset (>+3) compared to state-of-the-art gloss-free SLT methods. Furthermore, our approach also achieves competitive results on the PHOENIX14T dataset when compared with most of the gloss-based methods. Our code is available at https://github.com/zhoubenjia/GFSLT-VLP.

1. Introduction

Gloss-based SLT is limited by costly gloss annotation and information loss, motivating GFSLT-VLP’s gloss-free visual-language pretraining and end-to-end translation architecture. The method reports substantially higher BLEU-4 than prior gloss-free approaches while remaining competitive with gloss-based methods.

  • Gloss-based SLT requires labor-intensive specialist annotations and introduces an information bottleneck through intermediate gloss representations.Source: §1 · p. 2
  • VLP jointly aligns visual and textual representations while masked self-supervision trains the Text Decoder to capture sentence syntax and semantics.Source: §1 · p. 2
  • GFSLT transfers the pretrained Visual Encoder and Text Decoder into an end-to-end architecture that directly maps visual representations to spoken sentences without intermediate supervision.Source: §1 · p. 2
  • ≥+5 BLEU-4 on PHOENIX14T and ≥+3 on CSL-Daily over state-of-the-art gloss-free methods, without gloss annotations.Source: §1 · p. 2
  • The paper presents VLP as the first strategy, to its knowledge, for aligning visual and textual representations in a joint semantic space for gloss-free SLT.Source: §1 · p. 2

2. Related Works

Prior SLT work commonly uses sign glosses or recognition-derived representations, whereas gloss-free methods omit gloss supervision but face difficult cross-modal alignment. GFSLT-VLP is positioned as a way to narrow the performance gap between gloss-free and gloss-based translation.

  • Gloss-based SLT often uses continuous sign recognition to predict glosses or obtain visual representations before translation.Source: §2.1 · pp. 2–3, §2 · p. 2
  • Gloss-based methods improve translation but depend on fine-grained, specialist gloss annotation and may constrain scalability through the intermediate representation.Source: §2.2 · p. 3, §2.3 · p. 3
  • Gloss-free SLT omits gloss supervision during pretraining, training, and testing, but its performance is generally lower because video and spoken-language orders differ substantially.Source: §2.3 · p. 3
  • Existing gloss-free approaches include CNN-RNN modeling, attention-based feature learning, and modules for word verification, sentence generation, and cross-modal reranking.Source: §2.3 · p. 3

3. Method

GFSLT-VLP combines visual-language pretraining with masked sentence reconstruction, then transfers the pretrained encoder and decoder into an end-to-end gloss-free translation model.

Gloss-Free Sign Language Translation

  • The second-stage GFSLT model initializes its parameters from the pretrained visual encoder and text decoder for direct video-to-sentence translation without gloss supervision.The transferred components are then used in an encoder-decoder-like architecture for gloss-free SLT.Source: §3 · p. 4, §3.1 · p. 5, §3.2 · p. 5

Visual-Language Pretraining

  • VLP jointly pretrains visual and textual encoders to align sign-video and sentence representations in a shared multimodal semantic space.The method uses CLIP-inspired video-text pairing and similarity learning to produce language-indicated visual features.Source: §3.1 · pp. 4–5
  • The visual encoder processes frames with shared 2D CNN layers, temporal convolutional blocks, and a Transformer encoder for short- and long-term dependencies.The Transformer encoder captures long-term temporal relationships after spatial and short-term temporal processing.Source: §3.1 · pp. 4–5
  • Masked self-supervised learning trains the text decoder to restore masked words while combining complementary pretraining signals for richer representations.The approach also applies strong video augmentation, including geometric, color-space, and temporal transformations.Source: §3.1 · p. 5

Gloss-Free Sign Language Translation

  • The decoder generates one word at a time from the encoder state, ending at <EOS>, while the full model is optimized with video-to-sentence cross-entropy.Sentence generation begins with <BOS> and uses Linear and Softmax layers to calculate p(S|V).Source: §3.2 · p. 5, §3.1 · p. 5

4. Experiments

Experiments evaluate GFSLT-VLP on PHOENIX14T and CSL-Daily using BLEU and ROUGE, showing strong gloss-free gains and competitive gloss-based performance. Ablations examine augmentation, fine-tuning, and training duration.

Comparison with State-of-the-art Methods

  • +5.7 BLEU-4 on PHOENIX14T Test and +3.2±0.1 BLEU-4 on CSL-Daily over gloss-free methods.On PHOENIX14T, the method also improves ROUGE by about +2.6 on Test; on CSL-Daily, ROUGE improves by about 2.2±0.2.Source: §4.3 · p. 7
  • GFSLT-VLP is highly competitive with most gloss-based PHOENIX14T methods and remains close to selected gloss-based CSL-Daily systems.The PHOENIX14T comparison includes SLRT and STMC-T, while CSL-Daily comparisons include SLRT and BN-TIN-Transf without several auxiliary training or model enhancements.Source: §4.3 · p. 7

Ablation Studies

  • Strong data augmentation during VLP raises PHOENIX14T Test BLEU-4 from 19.84 to 22.05, whereas lightweight augmentation yields only small gains.The reported increase is +2.2, while lightweight augmentation improves Dev/Test BLEU-4 by about +0.3/+0.1.Source: §4.4 · p. 7, Table 3 · p. 7
  • Unified fine-tuning of the visual embedding module and Transformer encoder gives a +2.23 gain, exceeding separate fine-tuning of either component.Separate fine-tuning changes are reported as -0.01 and +1.39, while decoder fine-tuning contributes at most 1 point.Source: §4.4 · p. 8, Table 4 · p. 8
  • Gloss-free training requires more than 100 epochs for satisfactory performance, while 80 pre-training epochs balance results across the two datasets.The study attributes the longer downstream training requirement to slower convergence without intermediate representations.Source: §4.4 · p. 8, Table 5 · p. 8

5. Qualitative Results

Qualitative comparisons show that both systems produce complete sentences, but VLP reduces keyword errors and improves named-entity translations. The gains include more accurate place names and months.

Qualitative Results

  • VLP more accurately translates named entities, including place names and months, than the baseline.The comparison is based on PHOENIX14T test videos.Source: Table 6 · p. 8, §5 · p. 9
  • The baseline is more error-prone on keywords, sometimes producing translations that differ drastically despite capturing general meaning.Both systems can understand the general meaning and produce complete sentences.Source: §5 · pp. 8–9

6. Conclusion and Future work

The work reduces the semantic gap between visual and textual representations through visual-language and masked self-supervised pretraining for gloss-free SLT. It finds that data scale and model parameters significantly affect performance, while identifying large-scale gloss-free pretraining as future work.

  • The proposed pretraining paradigm combines masked self-supervised learning with visual-language supervision to learn language-indicated visual representations for gloss-free SLT.Source: §6 · p. 9
  • Experiments show that both data scale and model parameters significantly affect the method’s performance.Source: §6 · p. 9
  • Further research is needed on pretraining with a large-scale SLT dataset without gloss annotations.Source: §6 · p. 9

A. More Implementation details

The GFSLT model processes padded variable-length sign videos through ResNet-based visual feature extraction and temporal modules that capture short-term dependencies.

B.1. Impact of Model Parameter Size.

For GFSLT, increasing network depth does not improve performance and instead causes more severe overfitting, likely because SLT data are limited.

  • Increasing the network depth to four layers exacerbates overfitting instead of improving results.Source: §B.1 · p. 11
  • The authors attribute the deeper-network overfitting to the limited scale of SLT data.Source: §B.1 · p. 11
  • A sufficiently large SLT dataset may alleviate the overfitting issue caused by adding network layers.Source: §B.1 · p. 11

B.2. Impact of Mask Rate.

The masking ablation uses BERT-style token replacement and identifies a 15% masking rate as optimal for BLEU-4. The supplied passages also describe alternative pretraining behavior and parameter emphasis.

  • 15% masking achieves the optimal BLEU-4 score in the masking-rate experiment.Source: §B.2 · p. 11
  • The token-masking strategy replaces 80% of selected tokens with [Mask], 10% with another token, and leaves 10% unchanged.Source: §B.2 · p. 11
  • The experiments place more emphasis on pretraining the Visual Encoder than the Text Decoder.Source: §B.2 · p. 11
  • Freezing the text encoder as a teacher for visual-encoder learning does not produce satisfactory pretraining results.Source: §B.3 · p. 11

B.4. Impact of Loss weight

The loss-weight coefficient λ has a relatively minor effect near the tested settings, but increasing it eventually reduces performance, motivating λ=0.1. Additional ablations show that VLP and strong augmentation work best when combined across both training stages.

B.4. Impact of Loss weight

  • Performance fluctuations stayed around ±0.1 as λ varied, but increasing λ eventually caused performance to decline, so λ was set to 0.1.The study examined λ as the weight controlling the loss contribution associated with the Text Decoder.Source: §B.4 · p. 13

B.5. Investigation VLP on CSL-Daily

  • On CSL-Daily, VLP improved translation performance, while adding strong augmentation only in Stage 2 without VLP reduced it.The ablation compared VLP with strong augmentation applied in Stage 1 or Stage 2.Source: §B.5 · p. 13
  • The best CSL-Daily result occurred when strong data augmentation was applied in both stages together with VLP.This pattern was consistent with the experiments on Phoenix14T.Source: §B.5 · p. 13

Related papers

Loading 2307.14768v1…