Source-linked AI summary
TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Ben Swift, Hanna Suominen, Hongdong Li
TL;DR
SLT models must translate continuous sign videos despite unclear gesture boundaries and the temporal ambiguity of frame-wise representations. TSPNet uses multi-scale segments with hierarchical attention to learn richer sign features, improving BLEU and ROUGE on a major SLT dataset while reducing reliance on gloss annotations.
Problem
Existing SLT models often use frame-wise features that neglect temporal information, while accurate gesture segmentation and gloss annotation are difficult to obtain.
Method
TSPNet represents videos with multi-scale temporal segments and hierarchically combines local semantic consistency with non-local video context through attention.
Results
TSPNet raises BLEU from 9.58 to 13.41 and ROUGE from 31.80 to 34.96 on RWTH-PHOENIX-WEATHER-2014T.
Takeaways & Limitations
The method learns more expressive sign features and relaxes the need for expensive gloss annotations in video sign language translation.
Takeaways & Limitations
Low-frequency words such as city names remain challenging, and facial expressions conveying sign extent are not explicitly modeled.
Abstract
from arXiv · showhide
Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with no clear boundaries in between. Existing SLT models usually represent sign visual features in a frame-wise manner so as to avoid needing to explicitly segmenting the videos into isolated signs. However, these methods neglect the temporal information of signs and lead to substantial ambiguity in translation. In this paper, we explore the temporal semantic structures of signvideos to learn more discriminative features. To this end, we first present a novel sign video segment representation which takes into account multiple temporal granularities, thus alleviating the need for accurate video segmentation. Taking advantage of the proposed segment representation, we develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet. Specifically, TSPNet introduces an inter-scale attention to evaluate and enhance local semantic consistency of sign segments and an intra-scale attention to resolve semantic ambiguity by using non-local video context. Experiments show that our TSPNet outperforms the state-of-the-art with significant improvements on the BLEU score (from 9.58 to 13.41) and ROUGE score (from 31.80 to 34.96)on the largest commonly-used SLT dataset. Our implementation is available at https://github.com/verashira/TSPNet.
1 Introduction
SLT must translate continuous sign videos into natural-language sentences despite syntactic differences, costly gloss annotations, and ambiguous gesture boundaries. TSPNet addresses these challenges by learning temporally informed features from multi-scale video segments and improves translation quality on RWTH-PHOENIX-WEATHER-2014T.
- Motivation: Bootstrapping SLT models translate directly from videos without gloss annotations, extending to broader sign-language resources.Gloss annotations require sign-language expertise and are difficult to acquire.
- Motivation: Frame-wise features capture spatial appearance but neglect temporal dependencies that distinguish signs with similar body poses.Motion blur, fine-grained details, and transitions make gesture boundaries difficult to infer.
- Approach: TSPNet learns sign representations from video segments at multiple granularities instead of single frames.The representation encodes both spatial appearance and temporal dynamics while reducing reliance on accurate gesture segmentation.
- Approach: TSPNet exploits local semantic consistency and non-local context to reduce ambiguity in noisy sign segments.Its temporal semantic pyramid organizes segments hierarchically and uses inter-scale and intra-scale attention.
- Results: 13.41 BLEU and 34.96 ROUGE improve over baselines of 9.58 BLEU and 31.80 ROUGE on RWTH-PHOENIX-WEATHER-2014T.The reported gains reduce reliance on expensive gloss annotations for SLT models.
2 Related Work
Related SLT work addresses gesture-to-language alignment either with costly gloss annotations or direct video-to-text bootstrapping. TSPNet instead models spatial and temporal gesture information from segments while using local and non-local temporal structure.
- Sign Word Recognition: Word-level sign language recognition often uses constrained vocabularies, limiting feature generalization to broader settings.Large-scale datasets have been reported to improve generalization ability.
- Sign Language Translation: Gloss-based SLT methods ease alignment between gestures and natural-language words but require expensive sign-language expertise.Bootstrapping methods avoid glosses by predicting translations directly from sign videos.
- Sign Language Translation: Frame-wise bootstrapping approaches overlook temporal dependencies because sign gestures span multiple continuous frames.TSPNet addresses this by learning video features from segments.
- Sign Language Translation: TSPNet jointly models spatial appearance and temporal dynamics through hierarchical local and non-local temporal structure.This design counteracts inaccurate sign-gesture segmentation while learning discriminative representations.
3 Temporal Semantic Pyramid Network
TSPNet represents continuous sign videos with overlapping segments at multiple temporal scales, then hierarchically combines local and non-local context to learn less ambiguous sign features. Its attention modules address inaccurate segmentation by enforcing local semantic consistency and using broader video context.
- 3 Temporal Semantic Pyramid Network: TSPNet extracts segment features rather than frame-wise features to encode both spatial appearance and temporal dynamics despite uncertain gesture boundaries.The encoder uses I3D features for video segments and feeds the learned representation to a Transformer decoder.
- 3.1 Multi-scale Segment Representation: Multi-scale segments complement fine-grained gesture details with the stronger contextual semantics of longer segments.Smaller segments can capture finer gestures, whereas larger segments provide more context but are less suited to short gestures.
- 3.2.1 Shared Positional Encoding: Shared positional embeddings encode segment positions across scales after videos are padded so each scale has the same number of segments.Sharing the embedding weights reduces parameters and can improve training efficiency and alleviate overfitting when data is limited.
- 3.2.2 Enforcing Local Semantic Consistency: Inter-scale attention aggregates a pivot segment with temporally containing neighbors from larger scales to produce locally consistent representations.The neighbors provide contextual clues while preserving the pivot’s gesture content through a containment relation.
- 3.2.3–3.2.4 Local and Non-local Video Semantics: Intra-scale self-attention enhances aggregated local features across regions, while joint learning additionally lets non-local context interact with local semantics.The joint variant includes all pivot segments in each extended neighborhood so broader context can help recognize local gestures and reduce ambiguity from inaccurate segmentation.
4 Experiments
TSPNet is evaluated on the RPWT dataset against bootstrapping baselines using BLEU and ROUGE-L, with analyses of qualitative outputs and design choices. It improves translation quality through multi-scale segment representations and hierarchical feature learning, while remaining limited on low-frequency words and facial expressions.
- Experiment Setup: RPWT provides 7,096 training, 519 validation, and 642 test videos from nine German Sign Language signers, with translations and a vocabulary of around 3k German words.The dataset is the only publicly available standard SLT dataset used for large-scale training and inference.
- Experiment Setup: BLEU-n measures n-gram precision, while ROUGE-L measures F1 based on the longest common subsequences between predictions and reference translations.BLEU-4 summarizes precision over 1-, 2-, 3-, and 4-grams.
- Quantitative Comparison: 39.80% relative BLEU-4 improvement and 9.94% relative ROUGE-L improvement over Conv2d-RNN raise scores from 9.58 to 13.41 and from 31.80 to 34.96, respectively.Both TSPNet-Sequential and TSPNet-Joint outperform the state-of-the-art model by a large margin.
- Qualitative Comparison: TSPNet produces more accurate or semantically faithful example translations than Conv2d-RNN, including a synonym that preserves the meaning of “rain.”The qualitative difference is not fully reflected by the adopted metrics.
- Model Analysis: Adding multi-scale segments progressively improves performance, whereas 24-frame segments slightly reduce it because they may contain multiple sign gestures.Sixteen-frame segments provide the largest single-scale improvement, while longer segments can violate local semantic consistency.
- Model Analysis: Non-structural aggregation methods perform worse than TSPNet-Single, indicating that semantic structures matter when combining multi-scale segment features.The comparison includes position-wise pooling, position-wise fully connected aggregation, and nonrestrictive attention.
- Model Analysis: Fine-tuning I3D, sharing positional-embedding weights, and avoiding recurrent units support performance or efficiency, while recurrent operations can require up to two orders of magnitude more training time.Without I3D fine-tuning, the best BLEU-4 score drops to 11.23; separate positional embeddings reduce BLEU-4 by 0.08.
- Limitations: Low-frequency words such as city names remain challenging, and facial expressions relevant to sign extent are not explicitly modeled.The authors identify these issues as limitations and future work targets.
5 Conclusion
The paper presents TSPNet for sign language translation without reliable gesture boundaries. Its hierarchical use of multi-scale segment semantics and inter- and intra-scale attention learns more expressive features and substantially relaxes reliance on gloss annotations.
- Conclusion: TSPNet uses semantic relevance among multi-scale video segments to mitigate inaccurate gesture segmentation when learning sign video features.The model applies inter-scale and intra-scale attention to noisy segmented video clips.
- Conclusion: The learned features capture richer sign video semantics, and experiments show large-margin gains over previous bootstrapping models.The approach substantially relaxes the requirement for expensive gloss annotations.
Broader Impact
Automated sign language interpretation could improve communication access and social inclusion, but trustworthy deployment remains constrained by safety concerns and the limited scope of current data.
- Automated sign language interpretation could facilitate communication access and inclusion for deaf and hard-of-hearing people.The paper connects these systems with broader participation in society and access to health, education, and economic resources.
- Highly accurate and trustworthy translation requires further studies and regulation before use in life-critical emergency or health care situations.The authors recommend automated translators as auxiliary tools rather than alternatives to human interpreters in these scenarios.
- The RWTH-PHOENIX-WEATHER-2014T dataset is biased toward TV weather forecasting, which may limit applicability to real-life use.The paper nevertheless describes it as the only existing large-scale sign language translation dataset.