PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-Stream Network for Sign Language Recognition and Translation

Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, Brian Mak

arXiv:2211.01367v2cs.CV

TL;DR

Existing sign-language systems often encode visually redundant RGB videos directly, while SLR and SLT also face limited parallel data and a visual-to-language gap. The paper introduces dual-stream models for RGB and keypoint heatmaps, connects the streams with multiscale and distillation techniques, and reports state-of-the-art performance across SLR and SLT benchmarks. The framework is extended to translation by attaching a translation network, but it remains dependent on keypoint-estimator accuracy and data quality.

  • Problem

    Direct RGB encoding can overlook key sign-language information, while limited parallel data and the visual-to-language gap challenge SLR and SLT.

  • Method

    TwoStream-SLR separately encodes RGB videos and heatmap-based keypoint sequences, using lateral connections, sign pyramid networks, auxiliary supervision, and self-distillation; TwoStream-SLT adds a translation network.

  • Results

    TwoStream-SLR and TwoStream-SLT achieve state-of-the-art performance on SLR and SLT across Phoenix-2014, Phoenix-2014T, and CSL-Daily.

  • Takeaways & Limitations

    The framework supports both gloss recognition and sign-to-text translation while incorporating hand, face, and upper-body keypoint information alongside RGB video.

  • Takeaways & Limitations

    The approach depends on keypoint-estimator accuracy, and inaccurate estimations may hurt performance; its data-driven results may also be affected by dataset biases.

Abstract

from arXiv · show

Sign languages are visual languages using manual articulations and non-manual elements to convey information. For sign language recognition and translation, the majority of existing approaches directly encode RGB videos into hidden representations. RGB videos, however, are raw signals with substantial visual redundancy, leading the encoder to overlook the key information for sign language understanding. To mitigate this problem and better incorporate domain knowledge, such as handshape and body movement, we introduce a dual visual encoder containing two separate streams to model both the raw videos and the keypoint sequences generated by an off-the-shelf keypoint estimator. To make the two streams interact with each other, we explore a variety of techniques, including bidirectional lateral connection, sign pyramid network with auxiliary supervision, and frame-level self-distillation. The resulting model is called TwoStream-SLR, which is competent for sign language recognition (SLR). TwoStream-SLR is extended to a sign language translation (SLT) model, TwoStream-SLT, by simply attaching an extra translation network. Experimentally, our TwoStream-SLR and TwoStream-SLT achieve state-of-the-art performance on SLR and SLT tasks across a series of datasets including Phoenix-2014, Phoenix-2014T, and CSL-Daily. Code and models are available at: https://github.com/FangyunWei/SLRT.

1 Introduction

The paper addresses SLR and SLT by augmenting RGB video modeling with keypoint-based domain knowledge and mechanisms for interaction between visual streams. TwoStream-SLT extends TwoStream-SLR with translation, achieving state-of-the-art results across several benchmarks.

  • Task context: SLR predicts gloss sequences from sign videos, whereas SLT generates spoken-language text, optionally using glosses as an intermediate representation.SLR uses weak sentence-level gloss annotations without known gloss temporal boundaries.
  • Motivation: RGB-only approaches can be vulnerable to background or signer mismatch, motivating explicit modeling of manual and non-manual sign-language elements.The proposed domain knowledge includes handshape, palm orientation, facial expressions, and body, head, mouth, eye, and eyebrow movements.
  • Method: TwoStream-SLR uses separate encoders for RGB videos and heatmap-based keypoint sequences generated by an off-the-shelf estimator.HRNet trained on COCO-WholeBody extracts face, hand, and upper-body keypoints because most datasets lack keypoint annotations.
  • Method: Bidirectional lateral connections, sign pyramid networks with auxiliary CTC supervision, and frame-level self-distillation enable interaction and learning across temporal scales.The sign pyramid network is designed to capture glosses with different temporal spans, while auxiliary supervision helps shallow layers learn meaningful features.
  • Extension to translation: TwoStream-SLT is formed by attaching a translation network to TwoStream-SLR, extending the framework from recognition to translation.The translation model uses the encoded visual features produced by the recognition framework.
  • Results: TwoStream-SLR and TwoStream-SLT achieve state-of-the-art performance on SLR and SLT across Phoenix-2014, Phoenix-2014T, and CSL-Daily.The reported result spans both tasks and multiple benchmark datasets.

2 Related Work

Related work covers conventional SLR and SLT encoders, keypoint-based approaches, and multi-stream architectures. TwoStream-SLR combines these directions by modeling RGB videos and keypoint sequences in separate streams.

  • SLR and SLT: Prior SLR systems use video encoders based on 3D-CNNs, 2D-CNNs, or 2D+1D CNNs, with both single-stream and multi-stream architectures.The paper positions its approach within established CNN-based visual encoding designs.
  • Keypoints: Keypoint-based SLR and SLT methods have used estimated coordinates for cropping, independent cue processing, MLP-recurrent models, or skeleton graphs.These approaches differ in whether keypoints guide RGB processing or are modeled directly.
  • This work: TwoStream-SLR uses separate streams for RGB videos and heatmap-represented keypoint sequences, linking the multi-stream and keypoint-based directions.Its overview includes a video encoder, keypoint sequence encoder, joint head, lateral connections, and sign pyramid networks.
  • Multi-stream networks: Multi-stream action-recognition models commonly combine modalities such as RGB and optical flow or use different video frame rates.Related SLR and SLT systems adapt multi-stream ideas to RGB, optical flow, or multiple sign-language cues.

3 Method

TwoStream-SLR uses separate RGB-video and keypoint-sequence encoders, then exchanges and combines their information through lateral connections, joint prediction, pyramid supervision, and self-distillation. TwoStream-SLT extends this recognition model with translation networks and joint recognition-translation supervision.

  • Dual Visual Encoder: TwoStream-SLR encodes RGB videos and keypoint heatmap sequences with separate S3D-based visual streams supervised by CTC loss.HRNet generates pseudo keypoints for the face, hands, and upper body because most datasets lack keypoint annotations.
  • Bidirectional Lateral Connection: Bidirectional lateral connections exchange information between corresponding intermediate feature maps of the two S3D streams.The connections use element-wise addition after matching feature-map resolutions.
  • Joint Head and Late Ensemble: A joint head concatenates outputs from both encoders, while late ensemble averages predictions from the video, keypoint, and joint heads before CTC decoding.The joint head has the same architecture as the individual heads and is itself trained with CTC loss.
  • Sign Pyramid Network: Each stream contains a sign pyramid network that fuses multilevel S3D features into P2 and P3 for frame-level gloss prediction with auxiliary CTC supervision.The pyramid combines top-down and lateral pathways, using transposed convolution to match temporal and spatial dimensions.
  • Frame-Level Self-Distillation: Frame-level self-distillation uses averaged gloss probabilities from three heads as pseudo-targets and minimizes KL divergence against individual-head predictions.This adds fine-grained supervision to sentence-level CTC training and distills late-ensemble knowledge into each stream.
  • TwoStream-SLT: TwoStream-SLT appends an MLP and translation network to each of the three recognition heads and jointly optimizes recognition and translation losses.At inference, predictions from the three translation networks are combined using multi-source translation ensemble fusion.

4 Experiment

The experiments evaluate TwoStream-SLR and TwoStream-SLT on three sign-language datasets, using WER for recognition and BLEU/ROUGE-L for translation. The proposed components improve recognition, while the complete models achieve state-of-the-art results across the reported benchmarks.

  • Datasets: The study evaluates SLR on Phoenix-2014, Phoenix-2014T, and CSL-Daily, while SLT uses Phoenix-2014T and CSL-Daily.All ablation studies use the Phoenix-2014T SLR task.
  • Evaluation Metrics: WER measures SLR, whereas BLEU and ROUGE-L measure SLT; lower WER is better, while higher BLEU and ROUGE-L are better.
  • Comparison with State-of-the-art Methods: TwoStream-SLR achieves new state-of-the-art SLR results, outperforming previous best methods by 1.6% on Phoenix-2014, 1.1% on Phoenix-2014T, and 6.7% on CSL-Daily.
  • Comparison with State-of-the-art Methods: TwoStream-SLT surpasses all previous methods in both Sign2Text and Sign2Gloss2Text settings on Phoenix-2014T and CSL-Daily.Sign2Text directly generates text, while Sign2Gloss2Text first predicts glosses and then translates them.
  • Ablation Study: The component ablation reaches 18.52% WER with auxiliary CTC losses, 18.36 WER with a joint head, and 17.72 WER after adding frame-wise self-distillation.The reported component study is conducted on the Phoenix-2014T SLR task.
  • Ablation Study: Bidirectional lateral connections, sign-pyramid supervision, and self-distillation are evaluated as mechanisms for interaction, multi-temporal learning, and frame-level supervision.Self-distillation performs best with loss weight 1.0, ensemble pseudo-targets from three heads, supervision on all three heads, and soft targets.
  • Ablation Study: The keypoint encoder studies combinations of 79 HRNet keypoints covering upper body, hands, mouth, and other sign-relevant visual signals.The keypoints are generated using HRNet trained on COCO-WholeBody.

5 Conclusion

The paper introduces TwoStream-SLR and extends it to TwoStream-SLT by combining RGB and keypoint streams with interaction and auxiliary-training techniques. Both models achieve state-of-the-art performance across multiple sign-language benchmarks.

  • 5 Conclusion: TwoStream-SLR models RGB videos and keypoint sequences in separate streams to introduce sign-language domain knowledge into recognition.The framework uses bidirectional lateral connections, sign pyramid networks, and frame-level self-distillation to enable stream interaction.
  • 5 Conclusion: TwoStream-SLT extends TwoStream-SLR for translation by attaching an MLP and a translation network.
  • 5 Conclusion: TwoStream-SLR and TwoStream-SLT achieve state-of-the-art performance on SLR and SLT across Phoenix-2014, Phoenix-2014T, and CSL-Daily.
  • 5 Conclusion: CTC training marginalizes over feasible frame-level gloss alignments to compute p(G|V).The loss is applied to outputs from the video, keypoint, joint, and sign-pyramid heads.
  • 5 Conclusion: Frame-level self-distillation uses averaged predictions as pseudo-targets and KL-divergence terms to align stream outputs.The output length in the formulation is T/4.
  • 5 Conclusion: The translation objective is a sequence-to-sequence cross-entropy loss over the spoken-language sentence.

B.1 TwoStream-SLR

The TwoStream-SLR training and inference pipeline uses separately pretrained RGB and keypoint encoders, synchronized augmentation, and CTC decoding with fused head predictions.

  • B.1 TwoStream-SLR: The S3D backbone is pretrained on Kinetics-400 before separate CTC training of the video and keypoint encoders.The pretrained encoders are then loaded into TwoStream-SLR, while sign-pyramid networks and the joint head are randomly initialized.
  • B.1 TwoStream-SLR: Spatial cropping and frame-rate augmentation are applied to RGB videos and heatmap sequences while preserving spatial and temporal consistency.
  • B.1 TwoStream-SLR: Training uses the first four S3D blocks, freezes the first block, and runs for 40 epochs on 8 Nvidia V100 GPUs.The initial learning rate is 1e−3 with cosine annealing, Adam, and weight decay 1e−3.
  • B.1 TwoStream-SLR: Sign-pyramid networks are dropped during inference, and a CTC decoder produces the final gloss predictions.
  • B.1 TwoStream-SLR: The video, keypoint, and joint head probabilities are averaged before beam-search decoding with beam width 5.

B.2 TwoStream-SLT

TwoStream-SLT attaches translation modules to the three TwoStream-SLR heads and combines their independent predictions during decoding.

  • B.2 TwoStream-SLT: TwoStream-SLT attaches an MLP and translation network to each video, keypoint, and joint head.The translation networks use mBART-large-cc25 and are pretrained on gloss-text pairs before SLT training.
  • B.2 TwoStream-SLT: The model tokenizes text into subword tokens using the sentence-piece model associated with mBART.
  • B.2 TwoStream-SLT: During inference, predictions from the three translation networks are averaged to obtain each next-token probability.The networks correspond to the video, keypoint, and joint heads.
  • B.2 TwoStream-SLT: Text sequences are decoded with beam search using beam width 5 and no length penalty.

B.3 Keypoint Illustration.

The keypoint representation contains 79 points covering the face, hands, and upper body.

  • B.3 Keypoint Illustration.: The keypoint representation contains 26 face keypoints, 42 hand keypoints, and 11 upper-body keypoints.Together, these categories total 79 keypoints.
  • B.3 Keypoint Illustration.: The ablation table compares SingleStream-SLT with TwoStream-SLT variants using video, keypoint, or joint translation heads.It also includes independently seeded variants for some single-head configurations.

C More Experiments

The experiments examine translation variants and component effects, finding that the two-stream design generally matches or improves upon a single-stream baseline.

  • TwoStream-SLT variants using video, keypoint, or joint heads generally achieve comparable or better performance than the SingleStream-SLT baseline.The comparison evaluates variants with one translation network attached to each respective head.
  • The translation ablation also evaluates prediction fusion by averaging models trained with different random seeds on the same head.These fused variants are denoted TwoStream-SLT-V∗/K∗/J∗.
  • Additional ablation studies assess the effects of each proposed TwoStream-SLR component on Phoenix-2014 and CSL-Daily.The evaluation metric for this component study is WER in percent.

D Qualitative Results

Qualitative analyses visualize HRNet keypoint heatmaps and compare gloss predictions from video, keypoint, and combined streams across Phoenix-2014T and CSL-Daily.

  • HRNet heatmaps are reported as robust to signer appearance, hand positions, and palm orientation in most cases.The visualization samples five development-set frames from Phoenix-2014T and CSL-Daily.
  • Figure 5 pairs video frames with corresponding HRNet keypoint heatmaps, showing two rows from each of Phoenix-2014T and CSL-Daily.The figure organizes visual examples by dataset.
  • Using both video and keypoint streams yields more accurate gloss predictions than using either stream alone.The qualitative comparison covers samples from Phoenix-2014T and CSL-Daily.

E Broader Impact and Limitations

The paper frames TwoStream-SLR and TwoStream-SLT as tools for communication between hearing people and the deaf community, while noting data, reliability, bias, and keypoint-estimation limitations.

  • The proposed systems target improved communication between hearing people and the deaf community.
  • Existing SLR and SLT datasets contain only thousands of parallel data, which may make training insufficient for data-driven deep learning.
  • The authors warn that translation systems may have unpredictable failures and should not be used where failures could cause serious consequences.
  • Because the method is data-driven, performance may be affected by biases in the data and the data-collection process.
  • The approach relies on a keypoint estimator, so inaccurate estimations may hurt recognition and translation performance.The paper identifies improving the keypoint estimator as a potential way to facilitate both tasks.
Loading 2211.01367v2…