Source-linked AI summary
Fully Convolutional Networks for Continuous Sign Language Recognition
Ka Leong Cheng, Zhaoyang Yang, Qifeng Chen, Yu-Wing Tai
TL;DR
Continuous SLR must learn spatial and temporal patterns from weakly annotated videos, but existing CNN-RNN systems can struggle with unseen sequences and complicated training. This paper introduces an end-to-end fully convolutional network with jointly trained gloss feature enhancement for online recognition. The method achieves state-of-the-art performance on RGB-based methods across the reported datasets and shows consistent recognition in real-world scenarios.
Problem
Continuous SLR requires spatial-temporal learning from sentence-level annotations, while CNN-RNN hybrids struggle with unseen sequence patterns and non-trivial training.
Method
The paper proposes an end-to-end fully convolutional network with a jointly trained GFE module that enhances gloss features and sequence alignment learning.
Results
The proposed network achieves state-of-the-art RGB-based performance on both datasets, including 23.7% development and 23.9% testing WER on RWTH.
Takeaways & Limitations
The FCN shows consistent recognition in real-world scenarios where LSTM-based methods mostly fail, supporting online recognition applications.
Takeaways & Limitations
Stronger supervision using letter-sign gloss combinations remains a future direction requiring additional labeling preprocessing and professional sign-language knowledge.
Abstract
from arXiv · showhide
Continuous sign language recognition (SLR) is a challenging task that requires learning on both spatial and temporal dimensions of signing frame sequences. Most recent work accomplishes this by using CNN and RNN hybrid networks. However, training these networks is generally non-trivial, and most of them fail in learning unseen sequence patterns, causing an unsatisfactory performance for online recognition. In this paper, we propose a fully convolutional network (FCN) for online SLR to concurrently learn spatial and temporal features from weakly annotated video sequences with only sentence-level annotations given. A gloss feature enhancement (GFE) module is introduced in the proposed network to enforce better sequence alignment learning. The proposed network is end-to-end trainable without any pre-training. We conduct experiments on two large scale SLR datasets. Experiments show that our method for continuous SLR is effective and performs well in online recognition.
1 Introduction
Continuous SLR must handle spatial and temporal variation in weakly structured signing sequences, while prior CNN-RNN systems struggle with unseen sequence patterns and costly training. The paper proposes an end-to-end fully convolutional network with gloss feature enhancement for online recognition.
- 1 Introduction: Continuous SLR requires learning spatial and temporal information from signing sequences with similar gloss gestures and variable signing speeds.Different signers can produce the same gloss at different lengths, while similar movements complicate gloss discrimination.
- 1 Introduction: CNN-RNN hybrids often learn sequential patterns from seen signing sequences rather than glosses, limiting recognition of unseen sequence patterns.Their recurrent structure can make recognition sensitive to sequence order.
- 1 Introduction: Training prior hybrid models is generally non-trivial because many require pre-training and iterative training strategies.These requirements substantially lengthen the training process.
- 1 Introduction: The proposed fully convolutional network jointly learns spatial and temporal features and is trainable end-to-end without pre-training.The FCN is designed for continuous SLR and online recognition.
- 1 Introduction: The GFE module enhances feature representativeness, while the FCN recognizes unseen signing sentences, phrases, and paragraphs.Experiments evaluate the method on two public continuous SLR datasets.
2 Related Work
Continuous SLR differs from isolated SLR because glosses are unsegmented and only sentence-level annotations are available. Existing approaches estimate boundaries or use sequence-to-sequence frameworks, including CNN-RNN models for implicit alignment learning.
- 2 Related Work: Isolated SLR assumes glosses are temporally segmented, whereas continuous SLR recognizes glosses without segmentation from sentence-level annotations.The lack of gloss boundaries makes learning individual glosses more difficult under weak supervision.
- 2 Related Work: Early continuous SLR methods estimate gloss boundaries using video-region mining, tracking, HMMs, voting, or weakly supervised metric learning.These approaches infer temporal alignments before applying recognition methods.
- 2 Related Work: Sequence-to-sequence approaches adopt encoder and CTC frameworks from speech, handwriting, and video captioning recognition.The encoder extracts frame-sequence features while CTC learns the ground-truth gloss sequence.
- 2 Related Work: More recent methods use CNN-RNN hybrids to infer gloss alignments implicitly in continuous SLR.This extends sequence-to-sequence recognition frameworks to unsegmented signing sequences.
3 Method
The proposed network is fully convolutional, encoding frame-level spatial features and gloss-level temporal features before CTC decoding. A GFE module adds rectified, jointly trained supervision to improve gloss feature learning and alignment.
- Main stream design: The network maps input video frame sequences to target gloss sequences through a frame feature encoder, two-level gloss feature encoder, and GFE module.The frame encoder learns spatial features, while the gloss encoder models temporal information with convolutional layers.
- Main stream design: The frame encoder applies a convolutional backbone and global average pooling independently to each RGB frame, producing frame feature vectors.The input sequence has shape (t, c, h, w), and the output has one feature vector per frame.
- Main stream design: The two-level gloss encoder replaces recurrent temporal modeling with 1D convolutions that first model gloss semantics and then incorporate gloss-neighborhood information.Its sliding window spans multiple frame features, with the first-level window designed around the average signing-gloss length.
- Main stream design: The CTC decoder transforms encoded gloss features into prediction probabilities, considers possible alignments, and removes repeated and blank labels to produce the recognition result.The output includes an extra blank label, and decoding uses the many-to-one function B.
- Gloss feature enhancement: The GFE module searches the prediction map for the highest-probability alignment matching the ground-truth sequence, then uses paired gloss features and alignment labels for enhancement.The selected alignment is guaranteed to match the ground-truth sequence through the inverse CTC mapping B^-1.
- Gloss feature enhancement: The GFE branch uses weighted cross-entropy to reduce the effect of abundant blank labels, and joint training unifies the objective with the main stream.The blank-label loss is scaled by the balance ratio, while non-blank labels retain weight 1; joint training is described as more efficient than iterative training.
4 Experiments
Experiments on CSL and RWTH evaluate the proposed RGB-only FCN through implementation details, benchmark comparisons, ablations, and online-recognition scenarios. The results report strong accuracy, effective gloss-feature enhancement, and greater robustness than LSTM-based alternatives in tested settings.
- Experimental setup: The experiments use the CSL and RWTH-PHOENIX-Weather-2014 continuous SLR datasets, with word error rate as the evaluation metric.For CSL, Chinese characters are treated as words during evaluation.
- Benchmark results: 3.0% CSL WER and 23.7% development-set plus 23.9% test-set RWTH WER are reported as state-of-the-art results among RGB-based methods.On RWTH-PHOENIX Weather 2014 T, the reported development and testing WERs are 23.3% and 25.1%, respectively.
- Ablation studies: Both gloss-feature encoder levels are essential, because removing either causes WER to rise significantly.CNN-based temporal designs outperform LSTM counterparts on RWTH, whereas BiLSTM designs perform best on CSL in this ablation.
- Ablation studies: The GFE module with a balance ratio improves testing WER by 2.1% on RWTH and 5.2% on CSL.The larger CSL improvement is attributed to its richer spatial diversity, which makes spatial features harder to learn without GFE.
- Online recognition: Across four simulated online-recognition scenarios, the proposed network maintains consistent overall performance while the LSTM-based network degrades dramatically.The FCN also supports concatenated, split, and combined samples, with the reported results indicating reduced memory usage and recognition of phrases and paragraphs.
5 Conclusions
The paper concludes that its end-to-end fully convolutional network, jointly trained with a GFE module, achieves state-of-the-art benchmark performance and supports robust online continuous SLR. It also identifies stronger gloss supervision using letter-sign combinations as a future direction requiring additional labeling and professional knowledge.
- The proposed fully convolutional network is end-to-end trainable without pre-training for continuous SLR.
- A jointly trained GFE module enhances feature representativeness.
- The network achieves state-of-the-art performance on benchmark datasets using RGB-based methods.
- The authors identify stronger supervision from glosses composed of letter signs as a future research direction.This direction may require additional labeling pre-processing and professional sign-language knowledge.