Source-linked AI summary
SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting
Eunjee Choi, JungHoon Sung, Seongwhan Cho, Chu Xin, Younggeun Choi
TL;DR
CSLR lacks fine-grained temporal and semantic guidance because it relies on weak sequence-level supervision, while conventional video-text alignment is inefficient for small-batch video training. SMART addresses these issues with MLLM-guided alignment, temporal adaptation, and CSLR-guided spotting, achieving state-of-the-art performance across four sign-language benchmarks.
Problem
CSLR uses sentence-level gloss annotations and CTC, providing limited temporal and semantic guidance for fine-grained representation learning.
Method
SMART uses MLLM-generated motion descriptions for auxiliary video-text alignment, MSTA for temporal interactions, and CSFormer for CSLR-guided dense spotting.
Results
SMART achieves state-of-the-art performance across four sign languages, while ablations show recognition-derived gloss probabilities improve spotting.
Takeaways & Limitations
Recognition-derived gloss guidance and dense spotting supervision are complementary across recognition and spotting tasks.
Abstract
from arXiv · showhide
Continuous sign language recognition (CSLR) aims to recognize gloss sequences from unsegmented sign videos under weak sequence-level supervision. However, existing methods rely on sentence-level gloss annotations, providing limited temporal and semantic guidance for fine-grained representation learning. Conventional video-text alignment also requires large batch sizes, making it inefficient for memory-intensive sign language video training. In this work, we propose SMART, an MLLM-guided temporal alignment framework for joint sign recognition and spotting. SMART uses MLLMgenerated motion descriptions as auxiliary semantic cues and performs stable videotext alignment under small-batch training. To improve temporal representation learning, we introduce a Multi-Scale Temporal Adapter that models temporal interactions during transformer encoding. For dense temporal localization, SMART incorporates CSFormer, a CSLR-guided spotting module that injects recognition-derived gloss evidence into a boundary-aware spotting network. This unified framework enables CSLR features to benefit spotting, while spotting supervision complements weak CTC-based recognition. Experiments on four sign language benchmarks, including PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL datasets, demonstrate the effectiveness of SMART across both recognition and spotting tasks.
1 Introduction
SMART addresses limited temporal and semantic supervision in CSLR by unifying recognition with spotting, using MLLM-generated motion descriptions, temporal adaptation, and complementary dense localization guidance.
- Motivation: CSLR relies on sentence-level gloss annotations and CTC, which can produce peaky alignments that limit fine-grained temporal representation learning.Most frames may be assigned to the blank class while glosses are predicted in only a few frames.
- SMART framework: SMART uses MLLM-generated motion descriptions as auxiliary semantic cues and performs stable video-text alignment under small-batch training.The descriptions provide motion-aware language guidance for representation learning.
- SMART framework: The Multi-Scale Temporal Adapter captures multi-scale temporal dependencies within the CLIP backbone for temporal representation learning.MSTA models inter-frame interactions during transformer encoding while preserving pretrained CLIP representations.
- SMART framework: CSFormer integrates CSLR and spotting by injecting recognition-derived gloss guidance into dense frame-level localization and refining weak CTC-based alignments.This combines gloss-level recognition guidance with dense temporal supervision.
- Experiments: Experiments cover four benchmarks, including PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL, demonstrating complementarity across sign languages.The two Korean sign language datasets are used for CSLR and spotting for the first time.
2 Related Works
Related work shows that CSLR has progressed through CTC-based temporal modeling and pretrained vision-language adaptation, but still lacks rich frame-level and language-based guidance.
- Continuous Sign Language Recognition: CTC became a standard objective for CSLR because training provides sentence-level gloss annotations without ground-truth frame-level alignments.CTC supports end-to-end recognition under weak temporal supervision.
- Peaky Alignment in CSLR: CTC-based models commonly produce peaky alignments, predicting glosses at isolated time steps while assigning remaining frames to blanks.These alignments provide limited supervision for modeling fine-grained temporal dynamics.
- Recent CSLR methods: Recent methods improve CSLR through visual representation learning, temporal modeling, and adaptation of pretrained vision-language models such as CLIP.AdaptSign uses a frozen CLIP backbone with lightweight adaptation modules.
- Research gap: Existing CSLR methods still rely on sequence-level gloss supervision, while MLLM-generated language descriptions remain underexplored as auxiliary semantic cues.This gap motivates SMART's use of motion-aware descriptions for CSLR.
3 Methods
SMART jointly models continuous sign language recognition and sign spotting by combining MLLM-guided video-text alignment, multi-scale temporal adaptation, and CSLR-aware boundary refinement. Its components provide semantic supervision, temporal feature modeling, and recognition-guided localization within one framework.
- Framework overview: SMART combines MLLM-guided video-text alignment, a Multi-Scale Temporal Adapter, and the CSLR-aware CSFormer spotting module for joint CSLR and sign spotting.The alignment module supplies video-level semantic cues, MSTA enhances spatio-temporal features, and CSFormer injects recognition probabilities into boundary-aware spotting.
- MLLM-guided video-text alignment: Frame-level motion descriptions generated by LLaVA-OneVision-7B focus on hand movements and facial expressions and provide auxiliary semantic cues for video-text alignment.Descriptions are encoded with BERT, aggregated into a global text embedding, and used as auxiliary supervision rather than fused directly into the visual encoder.
- Multi-Scale Temporal Adapter: MSTA is inserted into the last four transformer blocks to model inter-frame temporal dependencies that frame-independent CLIP processing does not capture.It projects visual tokens into a low-dimensional temporal space, applies temporal operators at kernel sizes 3, 5, and 7, and adds the adapted feature through a residual connection.
- MLLM-guided video-text alignment: SigLIP applies an independent sigmoid loss to video-text pairs, enabling stable alignment optimization when memory-intensive CSLR training uses small batches.This avoids the large in-batch negative sets required by InfoNCE-based alignment.
- CSFormer sign spotting: CSFormer injects interpolated CSLR sequence logits into a separate spotting stream, exchanges information through bidirectional cross-attention, and predicts gloss transitions with stage-wise boundary heads.During spotting training, the recognition module is frozen; recognition-derived evidence guides boundary estimation while visual features refine sparse recognition spikes.
4 Experiments
SMART is evaluated across four sign-language benchmarks for recognition and spotting, with ablations, baseline comparisons, and qualitative analyses examining its components and outputs.
- Experimental Settings: Experiments use PHOENIX14-T, CSL-Daily, Large-scale KSL, and DS KSL; the KSL datasets support both CSLR and sign-spotting evaluation.PHOENIX14-T and CSL-Daily provide recognition benchmarks, while the KSL datasets additionally provide temporal boundary annotations.
- Recognition Results: SMART achieves the best WER across all four recognition benchmarks compared with prior methods.Table 2 reports WER, where lower values indicate better recognition performance, and excludes CSFormer from SMART recognition results.
- Spotting Results: CSFormer improves spotting by injecting CSLR-derived gloss guidance into dense temporal localization.SMART raises Test F1@50 from 7.37 to 96.72 on Large-scale KSL and from 5.90 to 59.77 on DS KSL while preserving or reducing WER.
- Ablation Study: On PHOENIX14-T, combining MSTA and alignment yields 17.58 Dev WER and 19.50 Test WER, outperforming the corresponding partial configurations.MSTA reduces Dev WER from 18.60 to 17.88, while the alignment objective reduces Test WER from 19.80 to 19.75 before joint application.
- Ablation Study: SigLIP improves Test WER to 19.75 under small-batch training, whereas InfoNCE increases it to 20.76 from the 19.80 no-alignment baseline.Batch size 8 is described as the practical upper bound in this training setting.
- Qualitative Analysis: Qualitative analyses show SMART produces spotting predictions closer to ground-truth boundaries and glosses while focusing more on hand movements and facial expressions.SMART reduces the deletion and substitution errors seen in baseline gloss predictions and attends less to background regions.
5 Conclusions
SMART unifies sign language recognition and spotting through semantic, temporal, and recognition-guided alignment. Across four benchmarks, its results support complementary benefits between gloss-level recognition guidance and dense temporal supervision.
- 5 Conclusions: SMART combines MLLM-generated motion descriptions, a Multi-Scale Temporal Adapter, and CSFormer for unified recognition and spotting.CSFormer injects recognition-derived gloss probabilities into boundary-aware spotting for dense frame-level localization.
- 5 Conclusions: CSFormer uses CSLR representations to provide gloss-level guidance for accurate sign localization.The module refines temporal boundaries while producing dense frame-level predictions.
- 5 Conclusions: Dense spotting supervision improves or preserves recognition performance depending on dataset characteristics.This establishes a complementary relationship between spotting supervision and recognition-derived guidance.
- 5 Conclusions: Experiments on four sign language benchmarks show state-of-the-art performance across sign languages.The evaluated benchmarks cover PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL.