PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

Hao Zhou, Wengang Zhou, Yun Zhou, Houqiang Li

arXiv:2002.03187v1cs.CV

TL;DR

CSLR models can overlook informative visual cues and their implicit collaboration. STMC addresses this with spatial cue decomposition, dual-path temporal modeling, and joint optimization, achieving state-of-the-art results across three CSLR benchmarks.

  • Problem

    CSLR models may focus on strong features while missing informative cues and the collaboration among hand, facial, and body visual information.

  • Method

    STMC uses an SMC module with self-contained pose estimation, a TMC module with intra-cue and inter-cue paths, and joint optimization for end-to-end sequence learning.

  • Results

    STMC surpasses previous state-of-the-art models on PHOENIX-2014, CSL, and PHOENIX-2014-T.

  • Takeaways & Limitations

    The framework learns spatial-temporal correlations among multiple visual cues for CSLR in an end-to-end fashion.

Abstract

from arXiv · show

Despite the recent success of deep learning in continuous sign language recognition (CSLR), deep models typically focus on the most discriminative features, ignoring other potentially non-trivial and informative contents. Such characteristic heavily constrains their capability to learn implicit visual grammars behind the collaboration of different visual cues (i,e., hand shape, facial expression and body posture). By injecting multi-cue learning into neural network design, we propose a spatial-temporal multi-cue (STMC) network to solve the vision-based sequence learning problem. Our STMC network consists of a spatial multi-cue (SMC) module and a temporal multi-cue (TMC) module. The SMC module is dedicated to spatial representation and explicitly decomposes visual features of different cues with the aid of a self-contained pose estimation branch. The TMC module models temporal correlations along two parallel paths, i.e., intra-cue and inter-cue, which aims to preserve the uniqueness and explore the collaboration of multiple cues. Finally, we design a joint optimization strategy to achieve the end-to-end sequence learning of the STMC network. To validate the effectiveness, we perform experiments on three large-scale CSLR benchmarks: PHOENIX-2014, CSL and PHOENIX-2014-T. Experimental results demonstrate that the proposed method achieves new state-of-the-art performance on all three benchmarks.

1 Introduction

CSLR seeks to translate sign sequences into gloss sentences, but models struggle to capture collaboration among diverse visual cues and may omit informative features. STMC addresses these challenges through spatial cue decomposition, dual-path temporal modeling, and joint optimization.

  • CSLR translates a series of signs into the corresponding sign gloss sentence.
  • Sign language combines manual hand elements with non-manual facial and upper-body cues whose temporal transitions can vary across glosses.
  • Existing multi-cue methods use external tools or inferred labels, limiting fully end-to-end differentiable learning.
  • Deep networks may prioritize strong features during convergence and omit other informative cues, limiting further performance improvement.
  • STMC decomposes spatial cues with a self-contained pose branch and models temporal relationships through intra-cue and inter-cue paths.
  • A joint optimization strategy enables end-to-end sequence learning, and experiments report performance surpassing prior state-of-the-art models on three CSLR benchmarks.

2 Related Work

Related work approaches CSLR through video representation, sequence learning, and multi-cue fusion. Multi-cue methods span multiple modalities and human-body semantics, using sensors, trackers, weak labels, or learned feature fusion.

  • CSLR systems generally combine video representation with sequence learning to map video sequences to sign gloss sequences.
  • Sequence-learning approaches include CNN-HMM systems, CTC-based methods for unsegmented inputs, and attention-based encoder-decoder models.
  • Multi-cue information is categorized into multi-modality and multi-semantic cues.
  • Multi-modality methods combine sensor-derived depth or infrared data with RGB or optical flow, while multi-semantic methods model body parts with different semantics.
  • Prior multi-semantic systems use segmented or tracked body-part features, hand patches, weak labels, and fusion with full-frame features.

3 Proposed Approach

The proposed approach is organized around spatial representation, temporal modeling, sequence learning, and joint loss optimization within an STMC framework.

  • The method presents an overall architecture followed by the SMC module, TMC module, sequence learning, and joint loss optimization.
  • The framework is designed to address spatial-temporal multi-cue sequence learning for CSLR.
  • Sequence learning and joint loss optimization complete the proposed STMC framework.

3.1 Framework Overview

For each video frame, STMC generates multiple spatial cue features, models their temporal relationships, and applies BLSTM-CTC processing for sequence learning and inference.

  • Given a video with T frames, CSLR predicts a corresponding sign gloss sequence with L words.
  • Spatial representation: The SMC module generates spatial features for full-frame, hands, face, and pose cues.
  • Temporal modeling: The TMC module captures intra-cue and inter-cue temporal correlations across different time steps and time scales.
  • Sequence learning: TMC outputs inter-cue and N intra-cue feature sequences, which BLSTM encoders and CTC layers use for sequence learning and inference.
  • SMC details: SMC estimates keypoints for face and hand patch cropping and outputs full-frame, hand, face, and pose features.

3.2 Spatial Multi-Cue Representation

The SMC module generates separate spatial representations for full-frame, hand, face, and pose cues, using differentiable pose estimation to guide feature extraction.

  • Spatial cue decomposition: The SMC module uses VGG-11 to generate spatial features for full-frame, hands, face, and pose cues.Its operations comprise pose estimation, patch cropping, and feature generation.
  • Pose estimation: Two deconvolutional layers and a point-wise convolution produce seven upper-body keypoint heat maps from VGG-11 feature maps.The deconvolutional layers upsample 14 × 14 maps to 56 × 56 resolution.
  • Pose estimation: A spatial softmax converts each heat map into a probability map, whose expected coordinates provide differentiable keypoint positions.The normalized predicted position of each keypoint is represented as Jk =(x̂, ŷ)k ∈[0, 1].
  • Feature generation: Predicted nose and wrist positions center face and hand crops, while pose coordinates and cropped visual regions are processed into cue-specific features.Hand crops are 24 × 24 and the face crop is 16 × 16; the two hands use weight-sharing convolutional layers.
  • Feature generation: The module is self-contained and differentiable, producing keypoint positions and visual-cue feature vectors for subsequent temporal modeling.The SMC mapping is denoted by Ωθ(·), with Jt,k representing the position of keypoint k at frame t.

3.3 Temporal Multi-Cue Modelling

The TMC module models temporal information through parallel intra-cue and inter-cue paths, preserving cue-specific features while learning their fused dynamics across time scales.

  • TMC design: The TMC module integrates spatiotemporal information through separate intra-cue and inter-cue paths rather than simple feature fusion.The intra-cue path captures unique cue features, while the inter-cue path combines fused features at different time scales.
  • TMC block: Each TMC block receives and produces an inter-cue feature matrix together with an intra-cue feature matrix concatenated across visual cues.The first input pair is o1=f1=[f1,1, f1,2, ···, f1,N].
  • Intra-Cue Path: The intra-cue path applies temporal transformations independently to cue features to provide unique representations at different time scales.The temporal transformation uses temporal convolution kernels, with k denoting temporal kernel size and C_N the number of output channels.
  • Inter-Cue Path: The inter-cue path temporally transforms features from the previous block and fuses information from the intra-cue path.A point-wise temporal convolution serves as a projection matrix between the two paths.
  • Implementation: Two TMC blocks are used, with temporal max-pooling after each block and temporal kernel size 5 for non-point-wise convolutions.Each path uses 1024 output channels, and max-pooling has stride 2 and kernel size 2.

3.4 Sequence Learning and Inference

The sequence-learning stage maps TMC feature sequences to sign glosses with a BLSTM and CTC, using joint losses during training and only the inter-cue path for inference.

  • Feature sequences: The TMC module produces inter-cue and intra-cue feature sequences that are used for sequence learning and inference.T′ denotes the temporal length of the final TMC output.
  • Recurrent sequence learning: A bidirectional LSTM maps the spatial-temporal feature sequence to hidden states and gloss-label probabilities over time.The hidden state at each time step is passed through a fully connected layer and softmax layer.
  • Connectionist Temporal Classification: CTC handles unknown alignments by summing probabilities over all alignment paths that map to the target gloss sequence.Its blank label represents stillness and transitions without precise meaning, while B removes blanks and repeated words.
  • Joint optimization: Training jointly optimizes the inter-cue CTC objective, auxiliary intra-cue CTC losses, and pose-estimation regression loss.The inter-cue path is primary; α balances the auxiliary intra-cue loss and β scales the pose regression loss.
  • Inference: During inference, only the inter-cue sequence and its BLSTM encoder generate gloss posteriors, which beam search decodes into the most probable sequence.Beam search searches within an acceptable range rather than enumerating every possible sequence.

4 Experiments

Experiments evaluate STMC on three CSLR datasets using pose-based keypoints, multi-cue inputs, staged training, and joint end-to-end optimization.

  • Datasets: STMC is evaluated on PHOENIX-2014, CSL, and PHOENIX-2014-T, covering German and Chinese continuous sign language recognition.PHOENIX-2014 contains 6,841 videos from nine signers, while CSL contains 5,000 videos across 100 sentences and 50 signers.
  • Pose preprocessing: Upper-body pose inputs comprise seven normalized keypoints represented by 2D pixel-coordinate tuples for every video frame.Keypoints are estimated with the publicly available HRNet toolbox.
  • Implementation details: The implementation uses four visual cues, with 1,024 output channels for inter-cue features and 256 for each intra-cue feature.Input frames are resized to 224×224, with shared-location random crops, 20% frame discarding, and random flips for augmentation.
  • Optimization: Training first uses staged optimization, then trains the full STMC network end-to-end with joint loss optimization.The initial stage trains a VGG11-based DNF network for pseudo-label decoding before STMC optimization.
  • Training and inference: Inference uses beam width 20, while training runs for 25 finetuning epochs followed by 30 end-to-end epochs on four GTX 1080Ti GPUs.The full-network stage uses Adam with learning rate 5 × 10−5 and batch size 2.

4.3 Framework Effectiveness Study

Module studies on PHOENIX-2014 examine STMC components, temporal paths, inference efficiency, and qualitative cue fusion.

  • Module analysis: The SMC module improves the baseline test performance by about 3% through multi-cue features.The baseline combines VGG11, 1D-CNN, and a BLSTM encoder.
  • Module analysis: Joint loss optimization adds 1.6% on the dev set and 1.7% on the test set compared with 1D-CNN.CTC loss guides the intra-cue path to learn each cue’s temporal dependency.
  • Module analysis: STMC reduces test-set WER by 4.8% relative to the baseline model.This result follows the combined spatial, temporal, and joint-optimization design.
  • Temporal paths: Among individual cues, pose performs worst, hand outperforms face, and full-frame features perform relatively better on the dev set.The inter-cue path leverages synergy among different cues.
  • Inference efficiency: The self-contained pose-estimation branch saves around 44% inference time versus an external VGG11-based pose model.The self-contained branch also shows slightly better performance than the off-the-shelf model.
  • Qualitative results: Qualitative results indicate that the inter-cue path learns correlations among multiple cues and produces a better prediction.Figure 5 compares outputs from different cues with estimated poses on the development set.

4.4 State-of-the-art Comparison

STMC is compared with prior methods on PHOENIX-2014, CSL, and PHOENIX-2014-T, achieving stronger reported results across these benchmarks.

  • PHOENIX-2014: On PHOENIX-2014, STMC achieves 20.7% WER on the test set, surpassing LS-HAN by 17.6%, CNN-LSTM-HMM by 5.3%, and DNF by 2.2%.The comparison includes multi-cue and RGB-plus-optical-flow competitors.
  • CSL: On CSL, STMC outperforms the best competitor by 4.1% on WER.The dataset evaluates signer-independent and unseen-sentence splits, with the latter described as more challenging.
  • PHOENIX-2014-T: On PHOENIX-2014-T, STMC surpasses all three evaluated combinations of CNN-LSTM-HMM.Those combinations use full-frame, hand, and mouth cues.

5 Conclusion

STMC learns spatial-temporal correlations among visual cues for CSLR through end-to-end multi-cue modeling. Its modules decompose spatial cues, preserve cue uniqueness, explore synergy, and support joint sequence learning.

  • The STMC framework learns spatial-temporal correlations among visual cues in CSLR end to end.
  • The SMC module uses a self-contained pose estimation branch to decompose spatial multi-cue features.
  • The TMC module combines intra-cue and inter-cue paths to preserve cue uniqueness while exploring synergy among cues.
  • A joint optimization strategy completes multi-cue sequence learning, and experiments are conducted on three large-scale CSLR datasets.
Loading 2002.03187v1…