Source-linked AI summary
Skeleton Aware Multi-modal Sign Language Recognition
Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, Yun Fu
TL;DR
SLR must recognize complex combinations of hand, body, and facial motion, but skeleton-based SLR is constrained by missing hand-keypoint annotations. SAM-SLR combines whole-body skeleton models with RGB and depth modalities, achieving first-place performance in both RGB and RGB-D challenge tracks.
Problem
SLR remains challenging because skeleton-based methods lack hand-keypoint annotations, despite sign language relying on complex hand, body, and facial motion.
Method
SAM-SLR uses pretrained whole-body pose estimators, SL-GCN for skeleton dynamics, SSTCN for skeleton features, and RGB and depth modalities in one framework.
Results
SAM-SLR achieves state-of-the-art performance on AUTSL and ranks first in both RGB and RGB-D tracks of the CVPR-21 isolated SLR challenge.
Takeaways & Limitations
Combining skeleton-based representations with RGB and depth modalities provides the framework's reported route to effective multimodal SLR.
Takeaways & Limitations
The authors retain late ensemble fusion because their early fused multi-stream implementation did not provide better performance, leaving early fusion for future work.
Abstract
from arXiv · showhide
Sign language is commonly used by deaf or speech impaired people to communicate but requires significant effort to master. Sign Language Recognition (SLR) aims to bridge the gap between sign language users and others by recognizing signs from given videos. It is an essential yet challenging task since sign language is performed with the fast and complex movement of hand gestures, body posture, and even facial expressions. Recently, skeleton-based action recognition attracts increasing attention due to the independence between the subject and background variation. However, skeleton-based SLR is still under exploration due to the lack of annotations on hand keypoints. Some efforts have been made to use hand detectors with pose estimators to extract hand key points and learn to recognize sign language via Neural Networks, but none of them outperforms RGB-based methods. To this end, we propose a novel Skeleton Aware Multi-modal SLR framework (SAM-SLR) to take advantage of multi-modal information towards a higher recognition rate. Specifically, we propose a Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics and a novel Separable Spatial-Temporal Convolution Network (SSTCN) to exploit skeleton features. RGB and depth modalities are also incorporated and assembled into our framework to provide global information that is complementary to the skeleton-based methods SL-GCN and SSTCN. As a result, SAM-SLR achieves the highest performance in both RGB (98.42\%) and RGB-D (98.53\%) tracks in 2021 Looking at People Large Scale Signer Independent Isolated SLR Challenge. Our code is available at https://github.com/jackyjsy/CVPR21Chal-SLR
1. Introduction
Sign language recognition is challenging because signs combine global body motion, delicate hand and arm gestures, and facial expressions, while performance varies across signers. SAM-SLR addresses missing hand-keypoint annotations by combining whole-body skeleton modeling with RGB and RGB-D modalities.
- SLR aims to support communication by automatically interpreting sign language, whose acquisition requires considerable learning and training.
- SLR must capture global body motion, delicate hand and arm gestures, facial expressions, and signer-specific variations simultaneously.Variations include speed, localism, handedness, and body shape.
- Skeleton-based methods offer information complementary to RGB but are limited by scarce controlled datasets and missing ground-truth hand annotations.
- SAM-SLR uses whole-body keypoints from pretrained pose estimators to construct an SLR-specific skeleton graph without extra annotation effort.
- SL-GCN models dynamics from the whole-body skeleton graph, while SSTCN exploits whole-body skeleton features and improves accuracy over traditional 3D convolution.
- SAM-SLR learns from six modalities and achieves state-of-the-art performance in AUTSL, ranking first in both RGB and RGB-D challenge tracks.
2. Related work
Prior SLR and action-recognition research uses deep video models, semantic guidance, skeleton representations, and multimodal fusion. However, existing methods remain limited in capturing complete motion information, while skeleton methods face annotation and data constraints.
- Recent SLR methods use RNNs, LSTMs, 3D CNNs, attention modules, and semantic detection or segmentation in two-stage pipelines.
- Existing SLR methods are still not effective enough to capture complete motion information.
- Skeleton-based action recognition models distinctive patterns from human joint positions and motion, either independently or alongside other modalities.
3. Our approach
The approach section introduces the skeleton models, RGB and depth baselines, and the multimodal SAM-SLR ensemble process.
- SL-GCN and SSTCN are introduced as models for skeleton keypoints and skeleton features, respectively.
- A baseline 3D CNN model is presented for the other modalities.
- SAM-SLR combines the modalities through a multimodal ensemble process.
3.1. SL-GCN
The SL-GCN models whole-body sign-language dynamics with a reduced spatio-temporal graph and multi-stream representations, while combining skeleton information with complementary visual modalities.
- Graph Construction and Reduction: Whole-body pose estimation provides 133 keypoints without requiring extra annotation effort for skeleton-based SLR.
- Graph Construction and Reduction: Graph reduction addresses noise and difficult long-range interactions in the 133-node whole-body skeleton graph, improving convergence and recognition rate.
- Graph Convolution: The spatio-temporal graph connects human-body keypoints spatially and models their dynamics with spatial partitioning and temporal convolution.
- Graph Convolution: Each SL-GCN block combines decoupled spatial convolution, cascaded spatial-temporal-channel attention, temporal convolution, and graph dropping.
- Multi-stream Approach: The multi-stream SL-GCN uses joint, bone, joint-motion, and bone-motion representations, training streams separately before weighted score combination.
- Multi-stream Approach: Early fusion did not improve performance in the authors’ implementation, so the framework uses late ensemble fusion and leaves alternatives for future work.
3.2. SSTCN
SSTCN recognizes sign language from whole-body pose features by processing down-sampled keypoint inputs with separable spatial-temporal convolutions and label smoothing.
- SSTCN: SSTCN uses features from 33 keypoints across 60 video frames, down-sampled to 24 × 24 with max pooling.
- SSTCN: The model replaces 3D convolution with separable 2D convolution to process whole-body features.
- Training: Label smoothing is adopted because one-hot cross-entropy labels can cause overfitting in some cases.
- Training: The smoothed-label loss combines penalties based on the predicted distribution’s difference from the real distribution and a uniform prior distribution.
3.3. 3D CNNs
The RGB, optical-flow, and depth-based modalities use 3D CNN baselines, with ResNet2+1D selected for its performance and controlled depth used to limit overfitting.
- 3D CNN Baselines: ResNet2+1D, which decouples spatial and temporal convolution sequentially, performs best among the evaluated popular 3D CNN architectures.
- 3D CNN Baselines: Increasing architecture depth does not improve performance and makes the network easier to overfit, motivating the choice of ResNet2+1D-18.
3.4. Multi-modal Ensemble
The framework ensembles outputs from four modalities, weighting each modality by validation accuracy before summing scores into the final prediction.
- Each modality produces class scores from its final fully connected layer before softmax, with one score per class.The number of classes is denoted by nc.
- Modalities are weighted according to validation accuracy and their scores are summed to produce the final predicted score.
- The ensemble selects final predicted classes using the indices of maximum combined scores.
- The authors found that their simple late ensemble outperformed early fusion and fully connected-layer ensemble alternatives.
4. Experiments
Experiments on AUTSL evaluate the proposed skeleton models, modality-specific baselines, ablations, and multimodal ensembles. The framework achieves strong validation and challenge-test performance, with skeleton streams complementing RGB and depth modalities.
- Dataset: AUTSL contains 38,336 clips from 43 signers performing 226 Turkish sign actions across 20 backgrounds.The dataset is divided into training, validation, and testing subsets.
- Ablation studies: Graph reduction is the most significant SL-GCN contributor, while augmentation, decoupling GCN, DropGraph, and STC attention also contribute to recognition rate.Without graph reduction, the model struggles to learn from the complex dynamics of a large skeleton graph; augmentation helps address overfitting.
- Multimodal ensembles: The skeleton-based SL-GCN and SSTCN ensemble outperforms RGB + Flow and Depth ensembles, while RGB and RGB-D multimodal ensembles further improve recognition rate.Depth flow is slightly more accurate than RGB flow because it introduces less noise.
- Baseline ablations: Label smoothing, swish activation, and CSL pretraining improve RGB 3D-CNN accuracy by 1%, 2%, and 1.4%, respectively.
- Challenge test set: After fine-tuning on the union of training and validation data, SAM-SLR surpasses baseline methods and ranks first in both RGB and RGB-D challenge tracks.The challenge-test predictions are reported with and without fine-tuning.
5. Conclusion
SAM-SLR combines whole-body skeleton modeling with RGB and depth modalities for sign language recognition. It uses pretrained pose estimators without extra skeleton annotation effort and achieves state-of-the-art challenge performance.
- SAM-SLR constructs a whole-body skeleton graph and uses SL-GCN to model embedded spatial and temporal dynamics without extra skeleton annotation effort.
- SSTCN exploits skeleton-feature information, while RGB and depth baselines are assembled with skeleton modalities in the multimodal framework.
- The assembled framework achieves state-of-the-art performance and wins both RGB and RGB-D tracks of the SLR challenge.