Source-linked AI summary
Continuous Sign Language Recognition with Correlation Network
Lianyu Hu, Liqing Gao, Zekang Liu, Wei Feng
TL;DR
CSLR methods often process frames independently, limiting their use of body trajectories that span adjacent frames. CorrNet adds correlation and identification modules to capture and emphasize those trajectories, achieving state-of-the-art accuracy across four large-scale datasets. The method is designed around a 2D-CNN feature extractor because the paper reports that 3D CNNs provide less precise gloss boundaries and lower accuracy.
Problem
Existing CSLR methods process frames independently and therefore fail to exploit cross-frame body trajectories important for identifying signs.
Method
CorrNet computes adjacent-frame correlation maps and uses an identification module to emphasize informative body trajectories without extra supervision.
Results
CorrNet achieves new state-of-the-art accuracy on PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL.
Takeaways & Limitations
Visualizations and comparisons show CorrNet attends to hand and face regions and outperforms prior spatial-temporal and hand/face-feature methods.
Takeaways & Limitations
The method uses a 2D-CNN feature extractor because the paper reports that 3D CNNs cannot provide as precise gloss boundaries and lead to lower accuracy.
Abstract
from arXiv · showhide
Human body trajectories are a salient cue to identify actions in the video. Such body trajectories are mainly conveyed by hands and face across consecutive frames in sign language. However, current methods in continuous sign language recognition (CSLR) usually process frames independently, thus failing to capture cross-frame trajectories to effectively identify a sign. To handle this limitation, we propose correlation network (CorrNet) to explicitly capture and leverage body trajectories across frames to identify signs. In specific, a correlation module is first proposed to dynamically compute correlation maps between the current frame and adjacent frames to identify trajectories of all spatial patches. An identification module is then presented to dynamically emphasize the body trajectories within these correlation maps. As a result, the generated features are able to gain an overview of local temporal movements to identify a sign. Thanks to its special attention on body trajectories, CorrNet achieves new state-of-the-art accuracy on four large-scale datasets, i.e., PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL. A comprehensive comparison with previous spatial-temporal reasoning methods verifies the effectiveness of CorrNet. Visualizations demonstrate the effects of CorrNet on emphasizing human body trajectories across adjacent frames.
1. Introduction
Body trajectories from hands, face, and posture are important cues for recognizing signs, but frame-independent CSLR methods miss cross-frame movement. CorrNet addresses this by dynamically correlating adjacent frames and emphasizing informative trajectories without extra supervision.
- Motivation: Hand, face, and body-posture trajectories provide salient cross-frame cues for identifying signs.These movements include rapid hand motion, finger twisting, facial expressions, head movements, and postural changes.
- Problem: Existing CSLR methods process frames independently, preventing them from exploiting adjacent-frame interactions and local temporal patterns.Their frame-wise features therefore fail to perceive hand and face movements that express a sign.
- Approach: CorrNet computes correlation maps between current and adjacent frames to capture trajectories across spatial patches.It dynamically attends to informative regions in both adjacent left and right frames.
- Approach: An identification module dynamically emphasizes body trajectories within the correlation maps, producing features that summarize local temporal movements.The procedure is end-to-end trainable without extra supervision such as body keypoints or heatmaps.
- Results: CorrNet achieves state-of-the-art accuracy on PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL.The paper also compares CorrNet with spatial-temporal reasoning methods and visualizes its attention to body trajectories.
2. Related Work
Prior CSLR research progressed from handcrafted and HMM-based systems to end-to-end CNN/RNN and CTC-based models, while later work enhanced temporal or spatial representations. CorrNet fits this progression by adding trajectory-focused correlation and identification modules to a standard CSLR pipeline.
- CSLR foundations: Earlier CSLR systems used handcrafted features or HMM-based temporal modeling to translate sentences step by step.CSLR translates frame sequences into glosses under weak supervision with sentence-level labels.
- End-to-end modeling: CNNs, RNNs, and CTC enabled end-to-end CSLR models that extract frame-wise features and model long-term temporal dependencies.CTC aligns target sentences with input frames during training.
- Recent advances: Later methods improved training efficiency or representations through iterative training, alignment losses, pseudo-labels, temporal feature refinement, and spatial attention.These approaches address feature-extractor training or emphasize informative spatial regions.
- Hand and face features: Several CSLR methods explicitly use hand and face information through multiple streams, pose-estimated keypoints, or cropped hand and face regions.Such approaches introduce additional visual inputs or pose-estimation processing.
- Correlation-based reasoning: Correlation operations have been used for geometric transformation, temporal object co-occurrence, dense patch correspondence, and optical-flow estimation.CorrNet applies this operation to cross-frame feature relationships in CSLR.
3. Method
CorrNet augments a 2D-CNN/1D-CNN/BiLSTM CSLR pipeline with correlation and identification modules that aggregate adjacent-frame trajectories and emphasize informative regions. The design uses local patch affinities, bidirectional temporal aggregation, and multi-scale spatial-temporal processing.
- Overview: The CSLR backbone uses a 2D CNN, 1D CNN, BiLSTM, and classifier to extract, temporally model, and decode gloss sequences.The 2D CNN produces frame-wise features; the 1D CNN and BiLSTM model short- and long-term dependencies.
- Overview: CorrNet places correlation and identification modules after each feature-extractor stage to incorporate cross-frame trajectories through a learnable residual contribution.The coefficient α is initialized at zero so the original feature behavior is initially preserved.
- Correlation Module: The correlation module computes dot-product affinities between a current-frame patch and neighboring patches in an adjacent frame.A K × K neighborhood limits computation and yields correlation maps of size H × W × K × K.
- Correlation Module: Sigmoid-centered affinities retain positive similarities and suppress redundant regions, while trajectory features aggregate neighboring-frame features weighted by those affinities.The correlation calculation is repeated across frames to track trajectories.
- Correlation Module: Bidirectional trajectories from the preceding and following frames are combined with a learnable coefficient β.This aggregates information from both temporal directions for each current-frame location.
- Identification Module: The identification module uses multi-scale dilated branches to locate informative body regions despite hand and face misalignment across frames.Progressive spatial-temporal dilation and grouped convolutions provide multiple receptive fields while reducing computation.
- Identification Module: Attention maps from the identification module multiply aggregated trajectories, emphasizing body motion and suppressing background or noise before residual integration.The output is x^out = x + α T(x) · M.
4. Experiments
Experiments evaluate CorrNet across datasets, ablations, alternative backbones, spatial-temporal reasoning methods, and approaches using hand or face features. Results show benefits from combining its modules, broad placement, and trajectory-focused attention.
- Ablation study: 18.8% and 19.4% accuracy are obtained on the PHOENIX14 Dev and Test sets when correlation and identification modules are combined.Using either module alone also improves accuracy, but the combined configuration performs better.
- Ablation study: 18.8% and 19.4% accuracy are achieved when CorrNet modules are placed after Stages 2, 3, and 4 on the PHOENIX14 Dev and Test sets.Combining multiple locations yields larger gains than using any single location.
- Generalizability: +2.0% and +2.2%, +2.0% and +2.0%, and +1.8% and +1.7% accuracy boosts are obtained with SqueezeNet, ShuffleNet V2, and GoogLeNet, respectively.These results support generalization across the tested backbones.
- Comparisons: CorrNet outperforms attention, temporal-reasoning, and hand-or-face-feature methods by emphasizing and aggregating body trajectories.The comparisons include SENet, CBAM, NLNet, I3D, R(2+1)D, TSM, and methods using additional hand or face supervision.
- State-of-the-art comparisons: 0.8% WER is achieved on CSL, while CorrNet also reports new state-of-the-art accuracy on CSL-Daily.The method’s visualizations show attention concentrated on human body regions, especially hands and face.
5. Conclusion
CorrNet captures body trajectories between adjacent frames and identifies body regions to improve continuous sign language recognition. Comparisons and visualizations support its effectiveness in attending to hand and face regions.
- CorrNet uses a correlation module for adjacent-frame trajectories and an identification module for locating body regions.
- Comparisons with spatial-temporal reasoning methods and methods using hand or face features demonstrate CorrNet’s superiority.
- Visualizations show CorrNet generally attends to hand and face regions to capture body trajectories.