Source-linked AI summary
SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, Houqiang Li
TL;DR
Hand gestures are central to sign language, yet current deep-learning SLR methods may face insufficient interpretability and overfitting with limited sign data. SignBERT self-supervises on hand poses using masking and reconstruction with model-aware hand prior, then fine-tunes for recognition. The authors report state-of-the-art performance across four benchmarks.
Problem
Deep-learning SLR methods may suffer insufficient interpretability and overfitting because sign data sources are limited.
Source: §1 · p. 1Method
SignBERT embeds both hands as visual tokens and pre-trains by masking and reconstructing them with a model-aware hand prior before fine-tuning for SLR.
Source: §5 · p. 8Results
SignBERT achieves state-of-the-art performance on NMFs-CSL, SLR500, MSASL, and WLASL.
Source: §1 · p. 2Takeaways & Limitations
The authors report that experiments validate SignBERT’s feasibility and effectiveness for downstream SLR.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Hand gesture serves as a critical role in sign language. Current deep-learning-based sign language recognition (SLR) methods may suffer insufficient interpretability and overfitting due to limited sign data sources. In this paper, we introduce the first self-supervised pre-trainable SignBERT with incorporated hand prior for SLR. SignBERT views the hand pose as a visual token, which is derived from an off-the-shelf pose extractor. The visual tokens are then embedded with gesture state, temporal and hand chirality information. To take full advantage of available sign data sources, SignBERT first performs self-supervised pre-training by masking and reconstructing visual tokens. Jointly with several mask modeling strategies, we attempt to incorporate hand prior in a model-aware method to better model hierarchical context over the hand sequence. Then with the prediction head added, SignBERT is fine-tuned to perform the downstream SLR task. To validate the effectiveness of our method on SLR, we perform extensive experiments on four public benchmark datasets, i.e., NMFs-CSL, SLR500, MSASL and WLASL. Experiment results demonstrate the effectiveness of both self-supervised learning and imported hand prior. Furthermore, we achieve state-of-the-art performance on all benchmarks with a notable gain.
1. Introduction
SignBERT addresses challenges in hand representation and data-limited SLR with a self-supervised framework that incorporates hand prior. The authors report state-of-the-art results on four benchmarks.
1. Introduction
- Hand gestures are central to sign language, but hand poses can be difficult to represent because hands are small, articulated, and prone to occlusion and pose-extractor failures.Pose-based methods are reported to lag behind RGB-based counterparts, while data-driven methods may suffer insufficient interpretability and overfitting due to limited sign data.Source: §1 · p. 1
- SignBERT treats each hand pose as a visual token, embeds gesture state, temporal order, and chirality, and pre-trains by masking and reconstructing tokens.The decoder incorporates hand prior to model hierarchical context; the pretrained encoder is fine-tuned with a prediction head for isolated SLR.Source: §1 · p. 2
- SignBERT achieves state-of-the-art performance on NMFs-CSL, SLR500, MSASL, and WLASL.The authors describe it as the first model-aware pre-trainable framework for sign language recognition.Source: §1 · p. 2
2. Relate Work
Prior SLR work uses RGB or pose inputs, while earlier pre-training work spans NLP, CV, and sign-language knowledge transfer. The paper positions SignBERT as addressing the lack of self-supervised pre-training for SLR and draws on hand-modeling methods for its hand prior.
2.1. Sign Language Recognition
- SLR approaches are grouped into RGB-based and pose-based methods, with CNNs, RNNs, GCNs, and Transformers used to model their inputs.Pose sequences are described as compact and semantic-aware; GCN-based methods show efficiency and promising performance.Source: §2.1 · p. 2
2.2. Pre-Training Strategy
- Prior pre-training work includes generic representations for NLP and CV, plus sign-language transfer from annotated or subtitled videos.The authors state that, to their knowledge, no prior work focuses on self-supervised pre-training for SLR.Source: §2.2 · p. 2
2.3. Hand-Modeling Technique
- Hand modeling spans geometric representations including meshes; the paper adopts MANO as a constraint in the pose decoder to import hand prior.MANO is described as a statistical model learned from high-quality hand scans and representing geometric changes in low-dimensional pose and shape spaces.Source: §2.3 · p. 2
2. Relate Work
- The framework overview depicts both-hand pose tokens, embedded with gesture state, temporal, and chirality information, entering self-supervised pre-training before downstream fine-tuning.Its pre-training uses masking strategies and a model-aware hand prior; the pretrained Transformer encoder is fine-tuned for recognition.Source: Figure 2 · p. 3
3. Our Approach
SignBERT processes both hands as embedded visual tokens, pre-trains by masking and reconstructing poses with a hand-model-aware decoder, and fine-tunes its encoder for SLR. Its objectives combine confidence-weighted pose reconstruction with regularization of the hand-model parameters.
3. Our Approach
- SignBERT has self-supervised pre-training and supervised fine-tuning stages, with both hands represented as visual tokens carrying gesture-state, temporal, and chirality embeddings.During pre-training, the decoder incorporates hand prior; for recognition, it is replaced by a prediction head.Source: §3 · p. 3
3.1. Framework Architecture
- Graph convolutions produce frame-level gesture semantics, while shared temporal embeddings and left/right chirality embeddings encode sequence order and hand identity for the Transformer.The hand-model-aware decoder maps encoder features to pose and camera parameters, and MANO turns predicted pose and shape into a hand mesh for projected 2D joints.Source: §3.1 · pp. 3–4
3.2. Pre-Training SignBERT
- Pre-training randomly selects 50% of tokens and applies joint masking, frame masking, or identity modeling with equal probability.Joint masking hides or disturbs selected joints; frame masking hides all joints in a token so reconstruction uses other frames or the other hand, while identity modeling leaves the token unchanged.Source: §3.2 · p. 4
3.3. Objective Functions in Pre-Training
- The pre-training objective combines hand reconstruction loss with regularization of MANO inputs.Reconstruction uses detected 2D poses as pseudo-labels, ignoring joints below a confidence threshold and weighting the remaining joints by confidence; regularization constrains parameter magnitude and change over time.Source: §3.3 · p. 5
3.4. Fine-Tuning SignBERT
- For downstream SLR, the hand-model-aware decoder is replaced by a prediction head trained with cross-entropy on unmasked hand-pose sequences.The paper also reports late fusion by summing hand-based predictions with predictions from full RGB data or full keypoints.Source: §3.4 · p. 5
4. Experiments
Experiments assess reconstruction feasibility, component choices, and downstream recognition across four public SLR benchmarks. The method outperforms prior methods across the reported benchmarks, with a 7.87% WLASL2000 improvement over the best competitor.
4.1. Datasets and Evaluation
- Four benchmarks span Chinese and American sign language; NMFs-CSL and SLR500 use per-instance accuracy, ASL datasets report Top-1/Top-5 per-class and per-instance accuracy, and pose feasibility uses PCK/AUC.MSASL and WLASL are web-collected and have unconstrained recording conditions and limited samples per word.Source: §4.1 · p. 5
4.2. Implementation Details
- Both hands are extracted from MMPose keypoints, cropped to 256 × 256, and 32 temporal frames are sampled for downstream training and testing.Training uses random sampling and testing uses center sampling.Source: §4.2 · pp. 5–6, §4.1 · p. 5
4.3. Ablation Study
- On STB, reconstructed joints consistently exceed masked inputs on PCK/AUC as masking increases; on HANDS17, reconstructed sequences also outperform masked inputs.The HANDS17 visualization shows ground-truth poses, masked inputs, reconstructions, and intermediate mesh results across six continuous frames.Source: Table 1 · p. 6, §4.3 · p. 6, Table 2 · p. 6
- Joint-level and frame-level masking each improve recognition over training without pre-training, and their combination reaches the best performance.The authors report Top-1 per-instance accuracy improvements of 13.08%, 12.95%, and 7.69% over the baseline.Source: §4.3 · p. 6, Table 4 · p. 7
- The model-aware decoder imports hand prior during pre-training, easing optimization and improving downstream isolated SLR while inflating 2D hand poses to 3D.Source: §4.3 · p. 6, Table 5 · p. 7
- Recognition accuracy peaks at three Transformer layers, while increasing the pre-training data ratio gradually improves downstream accuracy.The experiments use N = 3 by default.Source: §4.3 · p. 7, Table 6 · p. 7, Table 7 · p. 7
4.4. Comparison with State-of-the-art Methods
- 7.87% per-instance top-1 accuracy improvement over the best competitor on WLASL2000; Ours (H + R) leads NMFs-CSL and SLR500, with gains over pose- and RGB-based methods on MSASL.On SLR500, SignBERT reaches 97.6% top-1 accuracy; on WLASL, Ours (H + P) also outperforms the most challenging RGB-based method.Source: §4.4 · pp. 7–8, Table 3 · p. 7, Table 9 · p. 8
5. Conclusion
SignBERT is a self-supervised, pre-trainable SLR framework that uses hand-model-aware priors to capture hierarchical context. Experiments on four benchmarks report state-of-the-art performance with a notable margin.
- SignBERT achieves new state-of-the-art performance with a notable margin on four popular benchmark datasets.Source: §5 · p. 8
- During self-supervised pre-training, SignBERT masks and reconstructs hand-pose tokens, using a hand-model-aware decoder and masking strategies to capture hierarchical context.Source: §5 · p. 8
- SignBERT embeds both hands as visual tokens with gesture-state, temporal, and chirality information before fine-tuning the pre-trained framework for recognition.Source: §5 · p. 8