Source-linked AI summary
Video-based Sign Language Recognition without Temporal Segmentation
Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, Weiping Li
TL;DR
Continuous sign-language recognition is hindered by difficult temporal segmentation and costly isolated-word labeling. The paper proposes LS-HAN, combining two-stream CNN features, a temporal latent space, and hierarchical attention for segmentation-free sentence recognition. On the CSL dataset, it reports higher accuracy than both related LSTM methods and segmentation-based algorithms.
Problem
Continuous SLR commonly depends on difficult temporal segmentation and time-consuming isolated-fragment labeling.
Method
LS-HAN combines a two-stream CNN, a Latent Space model aligning video and sentence structure, and a Hierarchical Attention Network for recognition without temporal segmentation.
Results
LS-HAN achieves 0.141, 0.123, and 0.111 higher accuracy than CRF, LDCRF, and DTW-HMM, respectively, and 0.059 higher accuracy than LSTM-E.
Takeaways & Limitations
The framework demonstrates effective continuous SLR while avoiding temporal segmentation and sentence synthesis post-processing.
Takeaways & Limitations
The evaluation assumes approximate monotonically increasing video-sentence alignment because the dataset mainly contains simple single-clause sentences.
Abstract
from arXiv · showhide
Millions of hearing impaired people around the world routinely use some variants of sign languages to communicate, thus the automatic translation of a sign language is meaningful and important. Currently, there are two sub-problems in Sign Language Recognition (SLR), i.e., isolated SLR that recognizes word by word and continuous SLR that translates entire sentences. Existing continuous SLR methods typically utilize isolated SLRs as building blocks, with an extra layer of preprocessing (temporal segmentation) and another layer of post-processing (sentence synthesis). Unfortunately, temporal segmentation itself is non-trivial and inevitably propagates errors into subsequent steps. Worse still, isolated SLR methods typically require strenuous labeling of each word separately in a sentence, severely limiting the amount of attainable training data. To address these challenges, we propose a novel continuous sign recognition framework, the Hierarchical Attention Network with Latent Space (LS-HAN), which eliminates the preprocessing of temporal segmentation. The proposed LS-HAN consists of three components: a two-stream Convolutional Neural Network (CNN) for video feature representation generation, a Latent Space (LS) for semantic gap bridging, and a Hierarchical Attention Network (HAN) for latent space based recognition. Experiments are carried out on two large scale datasets. Experimental results demonstrate the effectiveness of the proposed framework.
Introduction
Continuous SLR is difficult because temporal segmentation is error-prone and isolated-word labeling is time-consuming. LS-HAN addresses these problems with segmentation-free recognition using two-stream features, latent-space modeling, and hierarchical attention.
- A two-stream 3D CNN generates global-local video feature representations for the framework.
- Temporal segmentation is non-trivial because diverse transitional movements are hard to detect, and inaccurate segmentation can propagate errors.
- Isolated-fragment labeling is highly time-consuming, limiting the practicality of segmentation-based continuous SLR pipelines.
- A Latent Space model explicitly exploits relationships between visual videos and text sentences to support HAN-based recognition.
- The proposed LS-HAN framework recognizes complete sentences without temporal segmentation and jointly optimizes relevance and recognition losses.
Related Work
Continuous SLR differs from isolated recognition and video description because it reconstructs sentence-level sign-language meaning. Prior pipelines rely on segmentation and language modeling, while latent-space methods address cross-modal relationships but may ignore temporal structure.
- Continuous SLR: Continuous SLR reconstructs sentence structures, unlike isolated SLR, which recognizes individual words or expressions.
- Continuous SLR: Existing continuous SLR pipelines typically perform temporal segmentation, isolated word or expression recognition, and sentence synthesis with a language model.
- Continuous SLR: Temporal segmentation is intrinsically difficult because transitional hand movements can be subtle and ambiguous, while inaccurate segmentation penalizes later steps.
- Video Description Generation: Video description generates brief appearance-oriented sentences, whereas continuous SLR provides semantic translations of sign-language sentences.
- Latent Space: Latent-space models bridge modalities, but a prior video-captioning embedding based only on Euclidean distance ignores temporal structures.
Signing Video Feature Representation
Signing videos require representations that handle hand-shape variation, occlusion, global motion, and local gestures in conventional RGB footage. The proposed two-stream 3D CNN combines complete frames with tracked hand patches to encode these complementary signals.
- Motivation: Hand gesture detection is challenged by large variations in hand shapes and orientations, as well as occlusions, especially in conventional RGB videos.
- Two-stream 3D CNN: The two-stream 3D CNN processes complete frames for global motion and cropped tracked hand patches for local gesture details.
- Two-stream 3D CNN: The streams use C3D-like structures with eight convolutional and five pooling layers, while two fully connected layers fuse global and local information.
- Two-stream 3D CNN: Left and right hand patches are concatenated as multi-channel inputs to the local stream.
- Feature Extraction: The CNN is pretrained on an isolated SLR dataset and used as a 4096-dimensional feature extractor over sliding 16-frame clips.
Proposed LS-HAN Model
LS-HAN performs continuous sign-language recognition without temporal segmentation by jointly learning video-sentence relevance in a latent space and sentence recognition with a hierarchical attention model. It combines global-local video features, structured sequence encoding, attention, and alignment-based relevance modeling.
- Joint LS-HAN learning: LS-HAN jointly optimizes video-sentence relevance in a latent space and recognition error from HAN.The objective minimizes both errors over paired videos and annotated sentences, with regularization and weighting terms.
- Latent-space representation: The framework maps global-local video features and one-hot sentence words into a common latent space.Transformation matrices project video and sentence representations into a latent space for semantic-gap bridging.
- Latent-space alignment: DTW measures video-sentence relevance by finding a minimum accumulating distance and a monotonic temporal warping path.Window-DTW restricts the computed distance matrix to windows to obtain bounded alignment paths.
- Hierarchical recognition: HAN hierarchically encodes clip sequences into word-level vectors, then word-level sequences into a latent representation for sentence decoding.Bidirectional LSTM encoders use attention, and a single LSTM decoder emits sentence words sequentially until the end symbol.
- Testing alignment reconstruction: During testing, alignment reconstruction is explored by splitting clips into two subsequences, pairs of adjacent clips, or seven evenly sized subsequences.Seven subsequences corresponds to the average sentence length in the training set.
Experiments
Experiments evaluate LS-HAN on two open-source continuous SLR datasets, comparing it with LSTM-based and segmentation-dependent methods, plus analyses of alignment, similarity, and parameter sensitivity.
- Datasets: Experiments use two open-source continuous SLR datasets: the proposed CSL dataset and RWTH-PHOENIX-Weather.The CSL recordings provide RGB, depth, and body-joint modalities from a Microsoft Kinect camera.
- Comparison with LSTM-based methods: LS-HAN achieves 0.059 higher accuracy than LSTM-E on the CSL dataset.The comparison highlights LS-HAN's temporal-information modeling during video-sentence correspondence optimization.
- Comparison with segmentation-dependent methods: LS-HAN obtains 0.141, 0.123, and 0.111 higher accuracy than CRF, LDCRF, and DTW-HMM, respectively.These comparison methods require segmentation during recognition.
- Comparison with recent methods: LS-HAN outperforms Deep Hand and Recurrent CNN in continuous SLR on RWTH-PHOENIX-Weather using hand sequences.The comparison describes LS-HAN as bridging the semantic gap with a latent space before sentence generation with HAN.
- Similarity analysis: HAN sentence probabilities and latent-space DTW distances show the expected increasing trend, despite some noise, across sampled video sequences.Ten videos each produce five likely sentences, whose probabilities and latent-space distances are compared.
- Sensitivity analysis: Extreme λ1 values produce high validation error, while the optimal choice is approximately 0.6.Recognition accuracy is evaluated on 2.5k validation instances while varying λ1.
Conclusion
The conclusion presents LS-HAN as a continuous SLR framework that removes temporal segmentation and sentence synthesis. It combines global-local video features with a temporally structured video-sentence latent space, while identifying longer compound sentences and real-time translation as future directions.
- Conclusion: LS-HAN eliminates error-prone temporal segmentation and sentence synthesis from continuous SLR.The framework is proposed for sentence-level recognition without those preprocessing and post-processing steps.
- Conclusion: Its two-stream 3D CNN combines global motion information with local gesture representations.The streams generate global-local video features for representation.
- Conclusion: The latent space optimizes labeled video-sentence distances and captures temporal structures by aligning frames to words.This latent-space representation connects signing videos with annotated sentences.
- Future work: Future work could extend LS-HAN to longer compound sentences and real-time translation tasks.These are identified as prospective application settings for the framework.