PaperlayerView exact revision on arXiv

Source-linked AI summary

SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding

Hezhen Hu, Weichao Zhao, Wengang Zhou, Houqiang Li

arXiv:2305.04868v1cs.CV

TL;DR

Sign language understanding is challenged by difficult hand representation, detector noise, limited sign data, and limited interpretability. SignBERT+ uses hand-pose visual tokens, model-aware self-supervised masked pre-training, and task-specific heads, achieving new state-of-the-art performance across isolated SLR, continuous SLR, and SLT. Its main scope boundary is dependence on the quality of off-the-shelf detected poses.

  • Problem

    Hand-based sign understanding is difficult because pose detectors fail under motion blur and complex backgrounds, while limited sign data contribute to over-fitting and insufficient interpretability.

  • Method

    SignBERT+ pre-trains on hand-pose visual tokens using gesture and spatial-temporal encoding, hierarchical joint-, frame-, and clip-level masking, and model-aware hand prior before task-specific fine-tuning.

  • Results

    SignBERT+ achieves new state-of-the-art performance across isolated SLR, continuous SLR, and SLT.

  • Takeaways & Limitations

    The framework provides one self-supervised, model-aware pre-training approach for multiple video-based sign language understanding tasks.

  • Takeaways & Limitations

    Performance remains limited by the quality of poses estimated by the off-the-shelf detector.

Abstract

from arXiv · show

Hand gesture serves as a crucial role during the expression of sign language. Current deep learning based methods for sign language understanding (SLU) are prone to over-fitting due to insufficient sign data resource and suffer limited interpretability. In this paper, we propose the first self-supervised pre-trainable SignBERT+ framework with model-aware hand prior incorporated. In our framework, the hand pose is regarded as a visual token, which is derived from an off-the-shelf detector. Each visual token is embedded with gesture state and spatial-temporal position encoding. To take full advantage of current sign data resource, we first perform self-supervised learning to model its statistics. To this end, we design multi-level masked modeling strategies (joint, frame and clip) to mimic common failure detection cases. Jointly with these masked modeling strategies, we incorporate model-aware hand prior to better capture hierarchical context over the sequence. After the pre-training, we carefully design simple yet effective prediction heads for downstream tasks. To validate the effectiveness of our framework, we perform extensive experiments on three main SLU tasks, involving isolated and continuous sign language recognition (SLR), and sign language translation (SLT). Experimental results demonstrate the effectiveness of our method, achieving new state-of-the-art performance with a notable gain.

1 INTRODUCTION

Sign language understanding spans isolated and continuous recognition and translation, but hand representation is difficult because poses are noisy and labeled sign data are limited. SignBERT+ addresses these challenges with self-supervised, model-aware pre-training and achieves state-of-the-art results across the three tasks.

  • Visual sign language understanding includes isolated SLR, continuous SLR, and SLT, with isolated SLR targeting word-level recognition.
  • Hand representation is difficult because hands are small, self-occluding, fast-moving, and observed against complex backgrounds.
  • Pose-based methods provide compact, semantic representations but suffer detector failures, while data-driven methods over-fit because sign data are limited.
  • SignBERT+ treats frame-wise hand poses as visual tokens, embeds gesture state and spatial-temporal position, and reconstructs masked tokens during self-supervised pre-training.
  • Joint-, frame-, and clip-level masking are combined with model-aware hand prior to capture hierarchical sign context, followed by task-specific prediction heads.
  • SignBERT+ achieves new state-of-the-art performance on isolated SLR, continuous SLR, and SLT.

2 RELATED WORK

Prior work studies video-based sign recognition and translation through RGB or pose representations, while self-supervised pre-training offers reconstruction-based alternatives to labeled learning. SignBERT+ combines these directions with hand-pose tokens and model-aware masked modeling.

  • Video-based sign language understanding comprises isolated SLR, continuous SLR, and SLT, each presenting distinct representation or sequence-alignment challenges.
  • RGB-based methods use CNN backbones, whereas pose-based methods employ CNNs, RNNs, or GCNs for structured hand-sequence modeling.
  • Self-supervised pre-training learns representations from unlabeled data through pretext tasks including prediction, contrastive learning, and reconstruction.
  • BERT-style masked modeling reconstructs masked tokens from context, but applying this idea to sign video is difficult because its spatial-temporal units lack clarified semantics.
  • Hand modeling methods represent shape using statistical or geometric formulations, including MANO, which learns low-dimensional pose and shape variation from hand scans.
  • SignBERT+ organizes pre-extracted hand poses as visual tokens, adding gesture and spatial-temporal encoding before masked pre-training and downstream fine-tuning.

3 OUR APPROACH

SignBERT+ uses a two-stage pipeline: self-supervised pre-training on hand-pose token sequences, followed by fine-tuning for downstream sign-language tasks. The framework reconstructs corrupted tokens while combining hierarchical masking with hand-aware context modeling.

  • The framework contains self-supervised pre-training followed by downstream-task fine-tuning.
  • Both hands are represented frame by frame as visual tokens, with gesture state and global spatial-temporal position information embedded in each token.
  • Pre-training reconstructs masked visual tokens from corrupted input sequences in a self-supervised paradigm.
  • Multi-level masking strategies are combined with an incorporated hand prior to model contextual information in sign sequences.

3.1 Framework Architecture

SignBERT+ represents hand pose sequences with gesture, spatial-temporal, and positional information, then uses a Transformer encoder and model-aware decoder to support task-specific prediction heads.

  • The architecture contains an input embedding layer, Transformer encoder, hand-model-aware decoder, and prediction head.
  • Input Embedding Layer: Hand pose graphs use spectral GCN processing and hierarchical pooling to derive frame-level gesture-state embeddings from both hands.Each hand’s 21 joints are pooled into six finger-and-palm subsets, then into one representation per frame.
  • Input Embedding Layer: Spatial-temporal encoding combines hand trajectory information from arm joints with temporal information before Transformer processing.
  • Transformer Encoder: The Transformer encoder applies repeated self-attention, feed-forward, and layer-normalization operations while preserving sequence feature size.
  • Hand-model-aware Decoder: The decoder extracts hand pose, shape, and weak-perspective camera parameters before differentiably reconstructing a hand mesh with MANO.The reconstructed mesh is projected through rotation and translation parameters, and additional task-specific prediction heads are designed for downstream tasks.

3.2 Pre-Training SignBERT+

SignBERT+ pre-training reconstructs masked continuous pose tokens to learn hierarchical context under joint, frame, clip, and identity corruption patterns.

  • Pre-training reconstructs masked visual tokens from corrupted pose sequences in continuous pose space rather than discrete word space.
  • Four equally likely masking operations model joint failures, frame failures, clip failures, and unchanged-token identity encoding.
  • Joint masking requires inference from remaining joints, while frame masking uses other frames and the other hand to capture temporal and mutual context.
  • Clip masking removes temporally continuous tokens, requiring the model to reconstruct them from existing motion patterns.
  • The objective combines masked-token reconstruction and regularization losses, filtering low-confidence joints and constraining MANO inputs toward plausible meshes.

3.3 Fine-Tuning SignBERT+

After pre-training, SignBERT+ adds task-specific heads for isolated recognition, continuous recognition, and translation, with optional RGB fusion because hand pose alone is insufficient.

  • Fine-tuning adds a task-specific prediction head above the pre-trained encoder and applies the corresponding supervised task loss.
  • Because hand pose alone is insufficient for full sign meaning, the framework provides RGB fusion variants alongside the vanilla pose-only model.
  • Isolated SLR: Isolated SLR uses attention-based temporal merging followed by an MLP-softmax classifier, with optional late fusion of RGB prediction scores.
  • Continuous SLR: Continuous SLR pools frame features and uses a CTC decoder to model unsegmented gloss sequences through many-to-one alignment paths.
  • Sign Language Translation: SLT reduces visual sequence length with pooling and a Transformer semantic modulator, then uses an autoregressive Transformer decoder with word and position embeddings.
  • Sign Language Translation: The translation decoder uses masked self-attention and cross-attention to combine generated words with modulated visual semantics, with optional cascaded RGB-pose attention.

4 EXPERIMENT

The experiments evaluate SignBERT+ across hand-pose pre-training and three downstream SLU tasks using established datasets and task-specific metrics.

  • Experiments assess framework feasibility on HANDS17, a video-level hand-pose dataset containing 292,820 frames from 99 sequences.
  • Downstream evaluation covers isolated SLR, continuous SLR, and SLT across MSASL, WLASL, SLR500, RWTH-Phoenix, and RWTH-PhoenixT.
  • Evaluation Metrics: Pose estimation is evaluated with PCK and AUC over PCK thresholds from 20 to 40 pixels.
  • Evaluation Metrics: Isolated SLR reports Top-1 and Top-5 per-instance and per-class accuracy, while continuous SLR uses Word Error Rate.
  • Evaluation Metrics: SLT is evaluated with BLEU and ROUGE-L F1-Score, measuring n-gram overlap and sentence-level structural similarity.
  • Implementation: Pre-training uses 230,246 videos collected from the evaluated datasets and additional sources, with Adam optimization for 60 epochs.

4.2 Ablation Study

The ablation studies validate reconstruction feasibility and examine architecture, masking, temporal encoding, positional encoding, decoder, and pre-training-scale choices. Results identify settings that improve downstream recognition, including complementary spatial-temporal encoding and combined multi-level masking.

  • Framework feasibility: Higher output metrics than corrupted inputs under joint, frame, and clip masking validate pose-reconstruction feasibility on HANDS17.The evaluation uses P@20, a PCK metric with a 20-pixel error threshold.
  • Architecture and temporal span: Three Transformer layers and a temporal clip span K=8 are selected because these settings reach peak MSASL accuracy in the corresponding studies.Further increasing the layer count does not add performance, while K=8 best captures temporal dynamics in masked clip modeling.
  • Temporal information extraction: Position encoding performs comparably to GCN-based temporal extraction, while the Transformer captures long-term sequential dependencies.The authors therefore use simple temporal position encoding as the temporal-order indicator.
  • Spatial-temporal position encoding: 10.62% per-instance Top-1 accuracy improvement over the full-set baseline comes from combining temporal and spatial position encoding.Spatial encoding adds global hand position information and the two encodings have complementary effects.
  • Masking strategy: Three masked modeling strategies improve full-set MSASL per-instance Top-1 accuracy by 5.13%, 6.92%, and 8.63%, respectively, with clip masking largest.Using all three strategies gives the best performance over the no-pre-training baseline.
  • Decoder and pre-training scale: The model-aware decoder improves downstream SLR over direct keypoint regression by regressing compact gesture embeddings and producing an intermediate 3D mesh.The decoder's hand prior eases optimization and adds an interpretable mesh representation.

4.3 Qualitative Visualization

Qualitative visualizations show reconstruction from corrupted or noisy hand poses, including missing joints and difficult hand interactions. The framework produces reconstructed poses and intermediate meshes aligned with RGB frames.

  • General cases: Under missing hand joints in several frames, the framework reconstructs poses by using contextual information from the sequence.The visualizations show input poses, reconstructed poses, and intermediate hand meshes on aligned RGB images.
  • General cases: The HANDS17 illustration compares ground-truth poses, masked inputs, reconstructed sequences, and middle mesh representations across 15 continuous frames.Blank regions indicate frames in which all joints were masked.
  • Interpretability: Intermediate hand meshes accompany reconstructed poses and improve the interpretability of the method.The mesh representation is shown alongside pose outputs in the qualitative visualizations.
  • Hard interaction cases: On hand-to-hand and hand-to-face interactions, the framework reconstructs poses despite ambiguity from similar appearance and mutual or self-occlusion.The qualitative samples visualize one hand's input, output, and intermediate mesh over five continuous frames.

4.4 Comparison with Other Pre-Training Strategies

The comparison evaluates supervised and self-supervised pre-training under matched backbone and data-volume settings across isolated SLR, continuous SLR, and SLT. SignBERT+ achieves the best performance across all downstream tasks with notable gains.

  • Overall comparison: SignBERT+ achieves the best performance with notable gains across isolated SLR, continuous SLR, and SLT.The evaluation covers MSASL, WLASL, SLR500, RWTH-Phoenix, and RWTH-PhoenixT tasks and datasets.
  • Supervised versus self-supervised pre-training: Supervised pre-training helps isolated SLR more than V-MoCo but performs worse than V-MoCo on continuous SLR and SLT.Its classification objective has limited generalization to downstream tasks with different objectives.
  • Data and evaluation setting: V-MoCo and SignBERT+ use no annotated data and scale with larger pre-training volumes, unlike the supervised comparison setting.The comparison uses the same embedding-layer and Transformer-encoder backbone for all methods.

4.5 Comparison with State-of-the-art Methods

SignBERT+ is evaluated against state-of-the-art methods across isolated SLR, continuous SLR, and SLT, with strong gains under challenging recording conditions. The framework narrows pose-based methods’ disadvantage and achieves reported improvements on isolated recognition and translation benchmarks.

  • SignBERT+ is compared with pose-based and RGB-based methods across isolated SLR, continuous SLR, and SLT.
  • Isolated SLR: 12.88% per-instance Top-1 accuracy improvement over SignBERT is reported on the full MSASL set.
  • Isolated SLR: Over 2% Top-1 per-instance accuracy improvement over the compared RGB-based methods is reported on all WLASL sets.
  • Isolated SLR: SignBERT+ greatly shrinks the performance gap between pose-based and RGB-based methods on isolated SLR and even outperforms challenging RGB-based methods in-the-wild.

4.6 Evaluation with Deaf Community

The deaf-community user study evaluates whether SignBERT+ preserves sign semantics under noisy pose inputs. Results indicate robust semantic preservation, while participant feedback identifies pose jittering and nonstandard gestures as directions for improvement.

  • The user study evaluates semantic preservation under different input noise levels using judgments from 10 deaf volunteers.Each volunteer judged whether the corresponding sign gloss could be identified from the framework output.
  • The output semantics remain well-preserved as input noise intensity increases, supporting the reported robustness of the framework.
  • The output correct rate is consistently better than the input correct rate across all noise levels.
  • Participants reported pose jittering and nonstandard gestures as reasons for failed recognition.
  • Design discussion: The discussion identifies token embedding, position encoding, masking strategy, and decoder or pre-training objective design as sign-specific design issues.
  • Limitations and future work: The detected-pose input quality remains a bottleneck, and jointly optimizing the pose detector with the framework is proposed as a possible solution.

5 CONCLUSION

The paper concludes that SignBERT+ combines hand-model-aware self-supervised pre-training with hierarchical masked modeling for three video-based sign language understanding tasks. It also notes that the technique is not intended to address potential privacy issues such as surveillance of sign language communication.

  • SignBERT+ is presented as a self-supervised pre-trainable framework incorporating hand prior for sign language understanding.
  • The framework treats both hands as visual tokens and embeds gesture state with spatial-temporal position information.
  • Hierarchical masked modeling at joint, frame, and clip levels is used to capture multi-level contextual information.
  • A hand-model-aware decoder and task-specific prediction heads support pre-training and fine-tuning across isolated SLR, continuous SLR, and SLT.
  • The technique is not intended to address potential privacy issues such as surveillance of sign language communication.
Loading 2305.04868v1…