PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language-Assisted Sign Language Recognition

Ronglai Zuo, Fangyun Wei, Brian Mak

arXiv:2303.12080v1cs.CV

TL;DR

Visually indistinguishable signs challenge vision-based sign recognition, though their glosses carry semantic information. NLA-SLR uses gloss semantics in label smoothing and inter-modality mixup, alongside a video-keypoint backbone. It achieves state-of-the-art performance on MSASL, WLASL, and NMFs-CSL.

  • Problem

    Visually indistinguishable signs are difficult for vision neural networks to recognize, while gloss semantics have been less considered in prior work.

    Source: §1 · pp. 1–2

  • Method

    NLA-SLR uses gloss semantics for language-aware label smoothing and inter-modality mixup, with a video-keypoint network that models RGB video and human keypoints across varied temporal receptive fields.

    Source: §1 · p. 2

  • Results

    NLA-SLR achieves state-of-the-art performance on MSASL, WLASL, and NMFs-CSL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework uses gloss semantics to promote sign language recognition and reports state-of-the-art performance on three widely used SLR datasets.

    Source: §1 · p. 2, §5 · p. 8

    The backbone relies on pre-extracted keypoints, and inaccurate keypoint estimation may hurt model performance.

    Source: E. Social Impact and Limitation · p. 15

Abstract

from arXiv · show

Sign languages are visual languages which convey information by signers' handshape, facial expression, body movement, and so forth. Due to the inherent restriction of combinations of these visual ingredients, there exist a significant number of visually indistinguishable signs (VISigns) in sign languages, which limits the recognition capacity of vision neural networks. To mitigate the problem, we propose the Natural Language-Assisted Sign Language Recognition (NLA-SLR) framework, which exploits semantic information contained in glosses (sign labels). First, for VISigns with similar semantic meanings, we propose language-aware label smoothing by generating soft labels for each training sign whose smoothing weights are computed from the normalized semantic similarities among the glosses to ease training. Second, for VISigns with distinct semantic meanings, we present an inter-modality mixup technique which blends vision and gloss features to further maximize the separability of different signs under the supervision of blended labels. Besides, we also introduce a novel backbone, video-keypoint network, which not only models both RGB videos and human body keypoints but also derives knowledge from sign videos of different temporal receptive fields. Empirically, our method achieves state-of-the-art performance on three widely-adopted benchmarks: MSASL, WLASL, and NMFs-CSL. Codes are available at https://github.com/FangyunWei/SLRT.

1. Introduction

Visually indistinguishable signs challenge vision-based recognition, while their glosses can carry useful semantic information. NLA-SLR uses gloss semantics in two training techniques and introduces a video-keypoint backbone; it achieves state-of-the-art performance on three SLR datasets.

1. Introduction

  • Visually indistinguishable signs (VISigns) share similar handshape and motion but can have varied meanings, and vision neural networks are less effective at recognizing them.Their glosses provide semantic information that prior work has taken less into consideration.Source: §1 · pp. 1–2
  • Language-aware label smoothing assigns higher soft-label weights to glosses with greater normalized semantic similarity to the ground-truth gloss.The weights are computed using fastText representations of glosses in the sign vocabulary.Source: §1 · p. 2
  • Inter-modality mixup combines vision and gloss features and uses corresponding mixed labels to promote separability among signs with distinct semantic meanings.The authors report that this technique significantly enhances discriminative power.Source: §1 · p. 2
  • The video-keypoint network models RGB video and human keypoints while deriving knowledge from sign videos with varied temporal receptive fields.Source: §1 · p. 2
  • NLA-SLR achieves state-of-the-art performance on MSASL, WLASL, and NMFs-CSL.Source: §1 · p. 2

2. Related Works

Prior sign-language recognition work commonly extracts visual features from RGB video, while some approaches also model keypoints. The paper situates its approach against these methods and discusses varied temporal receptive fields, gloss representations, vision-language models, and multi-label classification.

2. Related Works

  • Many recent SLR systems use CNN-based architectures such as I3D and R3D to extract RGB video features; this work uses S3D in VKNet for its accuracy-speed trade-off.Source: §2 · p. 2
  • Some SLR approaches jointly model RGB video and keypoints, but the cited methods require a dedicated keypoint network; VKNet represents keypoints as heatmaps so its encoder can share the video encoder architecture.Source: §2 · p. 3
  • Prior methods often train on fixed-length video clips, whereas VKNet uses varied temporal receptive fields to improve generalization capability.Source: §2 · p. 3
  • The paper uses fastText gloss representations and distinguishes its use of sign-label knowledge from vision-language models; it addresses VISigns in singly labeled SLR datasets through gloss language information.Source: §2 · p. 3

3. Methodology

NLA-SLR combines RGB video and keypoint inputs with gloss-derived supervision. Its framework comprises preprocessing, a video-keypoint network, and a head with language-aware label smoothing and inter-modality mixup.

3. Methodology

  • The framework processes video-keypoint pairs through VKNet and a head containing language-aware label smoothing and inter-modality mixup branches.Intra-modality mixup is also applied to RGB videos and keypoint heatmap sequences.Source: §3 · p. 3, §3.4 · p. 6

3.1. Data Pre-Processing

  • Each 64-frame RGB clip is paired with 63-keypoint heatmaps, then temporally cropped to a 32-frame pair for multi-receptive-field processing.The keypoints cover the upper body, mouth, and hands; the framework uses identical spatial and temporal augmentations for RGB and heatmaps.Source: §3.1 · p. 4

3.2. Video-Keypoint Network

  • VKNet uses two two-stream subnetworks for 32- and 64-frame video-keypoint pairs, concatenating video and keypoint features from each temporal field.Bidirectional lateral connections exchange information across video and keypoint streams and between same-modality encoders.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. Head Network

  • The head network contains a language-aware label smoothing branch and an inter-modality mixup branch.The branches produce semantic-similarity-weighted soft labels and blended vision-gloss features with corresponding labels.Source: Figure 5 · p. 4, §3.3 · p. 4

3.3.1 Language-Aware Label Smoothing

  • Language-aware label smoothing weights negative classes by gloss semantic similarity instead of distributing smoothing mass uniformly.FastText gloss features provide the similarities, and cross-entropy is applied to the resulting soft target.Source: §3.3.1 · p. 5

3.3.2 Inter-Modality Mixup

  • Inter-modality mixup adds mapped gloss features to a vision feature, then trains predictions against labels associated with each resulting feature.The auxiliary loss averages cross-entropy over the N inter-modality features; the mixup branch is dropped at inference, while FC2 parameters are integrated into FC1 during training.Source: §3.3.2 · p. 5

3.3.3 Overall Loss

  • The head-network loss sums classification loss and inter-modality mixup loss weighted by a trade-off hyperparameter.The head is applied independently to each vision feature, and the model loss sums the losses of those heads.Source: §3.3.3 · p. 5

3.4. Intra-Modality Mixup

  • Intra-modality mixup applies classical Mixup separately to RGB videos and keypoint heatmap sequences.The authors report that this regularization is helpful for sign language recognition.Source: §3.4 · pp. 5–6

4. Experiments

Experiments evaluate NLA-SLR on three sign-language datasets and report gains over prior best methods. Ablations examine its backbone, language-aware label smoothing, and mixup components.

4.1. Datasets and Evaluation Metrics

  • MSASL and WLASL provide multiple vocabulary-size subsets; NMFs-CSL has 1,067 glosses, but its index-only labels preclude inter-modality mixup and require vanilla label smoothing.The reported metrics are per-instance and per-class accuracy, except NMFs-CSL, for which only per-instance accuracy is reported.Source: §4.1 · p. 6

4.2. Implementation Details

  • Training uses pretrained video and keypoint encoders, consistent RGB/keypoint augmentations, 32-sample batches, and 100 epochs; three-crop inference averages predictions from start, middle, and end crops.Ablations use single-crop inference.Source: §4.2 · p. 6

4.3. Comparison with State-of-the-art Methods

  • NLA-SLR beats prior bests by 2.56%/2.50%/1.46% top-1 on MSASL 1,000/200/100, 1.87%/1.68% per-instance/class top-1 on WLASL, and 83.7% vs. 79.2% top-1 on NMFs-CSL.The WLASL comparison is against SAM-SLR-v2; the NMFs-CSL comparison is against BEST.Source: §4.3 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7

4.4. Ablation Studies

  • Two-stream VKNet models outperform single-stream video or keypoint encoders, 64-frame models outperform 32-frame models, and mixed temporal receptive fields outperform VKNet-64, especially on top-5 accuracy.The authors interpret the different temporal receptive fields as complementary sources of information.Source: §4.4 · p. 7, Table 4 · p. 7
  • Lang-LS improves VKNet by 1.22%/1.11% top-1/top-5, sign mixup raises top-1 from 57.19% to 60.32%, and both yield 61.05%/91.45% top-1/top-5.The combined model uses VKNet with both Lang-LS and sign mixup.Source: §4.4 · p. 7, Table 5 · p. 8
  • Lang-LS yields the largest gain for similar-semantic VISigns, while inter-modality mixup yields the largest gain for distinct-semantic VISigns.On WLASL2000, the reported top-1 changes are 50.50 to 64.36 for VS-S and 50.93 to 56.07 for VS-D.Source: §4.4 · p. 8, Table 9 · p. 8

5. Conclusion

The conclusion presents NLA-SLR as a gloss-informed sign recognition framework with language-aware smoothing, inter-modality mixup, and a video-keypoint backbone. It reports that the approach surpasses previous best methods on three benchmarks.

  • NLA-SLR surpasses previous best methods on MSASL, WLASL, and NMFs-CSL.Source: §5 · p. 8
  • Language-aware label smoothing generates soft labels using normalized semantic similarities among glosses to ease training.Source: §5 · p. 8
  • Inter-modality mixup blends vision and gloss features and labels to increase separability among signs with distinct semantic meanings.Source: §5 · p. 8
  • VKNet models RGB videos and human keypoints while drawing on sign videos with different temporal receptive fields.Source: §5 · p. 8

A. More Implementation Details

The appendix describes VKNet's bidirectional lateral connections and its selected keypoint inputs, and identifies an ablation comparing inter-modality mixup across head networks.

A. More Implementation Details

  • Bidirectional lateral connections exchange video-video, keypoint-keypoint, and video-keypoint information across the first four S3D blocks.Convolutions and transposed convolutions match spatial resolutions across video and keypoint streams and temporal resolutions within each stream.Source: A. More Implementation Details · p. 11
  • VKNet uses 63 HRNet-estimated keypoints per frame: 11 upper-body, 10 mouth, and 42 hand keypoints.Source: A. More Implementation Details · p. 11
  • Table 10 is an ablation of inter-modality mixup across different head-network types.Source: Table 10 · p. 11

B. More Experiments

The experiments examine inter-modality mixup across head groups, keypoint selection, lateral connections, backbone design, and gloss-feature representations. The reported comparisons favor applying mixup to all heads, using all three connection types, VKNet over SlowFast, inter-modality mixup over contrastive learning, and fastText among the tested word representations.

Head Choices for Inter-Modality Mixup

  • Applying inter-modality mixup to all head groups achieves the best performance, while applying it to any one group outperforms the baseline.Source: B. More Experiments · p. 11

Keypoint Selection

  • 21.37% top-1 accuracy is obtained with upper-body keypoints alone; adding hand keypoints improves top-1 accuracy by 27.17%.Mouth keypoints also have a positive effect; using 63 keypoints per frame comprises 11 upper-body, 42 hand, and 10 mouth keypoints.Source: B. More Experiments · p. 11, Table 11 · p. 11, Figure 7 · p. 12

Bidirectional Lateral Connections

VKNet vs. SlowFast

  • VKNet consistently outperforms SlowFast on all four metrics when compared under the described temporal-sampling setup.The comparison samples a 32-frame pair from the 64-frame input with a stride of two frames to mimic SlowFast.Source: B. More Experiments · p. 12, Table 13 · p. 13

Inter-Modality Mixup vs. Contrastive Learning

Word Representation Learning Methods

  • fastText performs better than word2vec, GloVe, and BERT-base on all four metrics; GloVe leads top-1 while word2vec leads top-5 among the first two.BERT-base performs worse than the other methods, which the passage attributes to its not being dedicated to word representation learning.Source: B. More Experiments · p. 13, Table 15 · p. 13

C. Visualization

The visualizations examine gloss-feature similarities and HRNet keypoint heatmaps. Similarities roughly reflect semantic relations, while the heatmaps appear robust across several signer and scene variations.

Gloss Feature Similarity

  • Gloss-feature cosine similarities roughly reflect semantic similarities, including high-similarity pairs such as “cold” and “winter.”Other examples include “adapt” and “change,” “city” and “community,” “speech” and “oral,” and “accent” and “voice.”Source: C. Visualization · p. 13

Keypoint Heatmaps

  • HRNet keypoint heatmaps are described as robust to signer appearance, background variation, hand position, and palm orientation.The visualizations use six randomly selected frames from three signers across WLASL2000, MSASL1000, and NMFs-CSL.Source: C. Visualization · p. 13

D. Qualitative Results

The qualitative examples show NLA-SLR distinguishing visually indistinguishable signs that VKNet misses, for both similar-meaning and distinct-meaning pairs. Figure 9 also presents sampled sign frames with their estimated keypoint heatmaps.

Qualitative Results

  • NLA-SLR distinguishes similar-meaning VISign pairs “doctor”/“nurse” and “calculate”/“multiply,” which the VKNet baseline fails to classify.For this Figure 10 comparison, NLA-SLR does not use intra-modality mixup; ground-truth glosses are highlighted in red.Source: D. Qualitative Results · p. 13
  • Figure 9 displays randomly selected sign frames alongside HRNet-estimated keypoint heatmaps.Source: Figure 9 · p. 14
  • NLA-SLR also recognizes distinct-meaning VISign pairs “champion”/“mushroom” and “friend”/“medicine.”The authors attribute these successes to language-aware label smoothing and inter-modality mixup.Source: D. Qualitative Results · p. 14

E. Social Impact and Limitation

The paper frames sign language recognition as a way to help bridge communication between hearing and deaf communities. It notes that data bias and reliance on pre-extracted keypoints constrain the method.

Social Impact

Limitation

Related papers

Loading 2303.12080v1…