Source-linked AI summary
SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting
Eunjee Choi, JungHoon Sung, Seongwhan Cho, Chu Xin, Younggeun Choi
TL;DR
CSLR relies on sentence-level gloss annotations, providing limited temporal guidance for fine-grained representation learning. SMART combines MLLM-guided video-text alignment, temporal modeling, and CSLR-guided spotting. Experiments on four sign-language benchmarks show state-of-the-art performance across sign languages.
Problem
CSLR's sentence-level gloss supervision provides weak temporal guidance, limiting dense frame-level representation learning and modeling of fine-grained sign dynamics.
Source: §1 · p. 2Method
SMART combines MLLM-generated motion descriptions for small-batch video-text alignment, a Multi-Scale Temporal Adapter, and CSFormer for CSLR-guided sign spotting.
Source: §1 · p. 3Results
SMART achieves state-of-the-art performance across sign languages on four sign-language benchmarks.
Source: §5 · p. 14Takeaways & Limitations
Results indicate complementarity between gloss-level recognition guidance and dense temporal supervision across different sign languages.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Continuous sign language recognition (CSLR) aims to recognize gloss sequences from unsegmented sign videos under weak sequence-level supervision. However, existing methods rely on sentence-level gloss annotations, providing limited temporal and semantic guidance for fine-grained representation learning. Conventional video-text alignment also requires large batch sizes, making it inefficient for memory-intensive sign language video training. In this work, we propose SMART, an MLLM-guided temporal alignment framework for joint sign recognition and spotting. SMART uses MLLMgenerated motion descriptions as auxiliary semantic cues and performs stable videotext alignment under small-batch training. To improve temporal representation learning, we introduce a Multi-Scale Temporal Adapter that models temporal interactions during transformer encoding. For dense temporal localization, SMART incorporates CSFormer, a CSLR-guided spotting module that injects recognition-derived gloss evidence into a boundary-aware spotting network. This unified framework enables CSLR features to benefit spotting, while spotting supervision complements weak CTC-based recognition. Experiments on four sign language benchmarks, including PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL datasets, demonstrate the effectiveness of SMART across both recognition and spotting tasks.
1 Introduction
SMART unifies continuous sign language recognition (CSLR) and sign spotting to address weak temporal supervision in CSLR. It combines MLLM-guided video-text alignment with dense spotting supervision and recognition-derived gloss guidance.
1 Introduction
- CTC-based CSLR often yields peaky alignments, while large-vocabulary spotting with accurate boundaries remains underexplored beyond retrieval, localization, and sparsely annotated approaches.The passage identifies prior spotting methods using short query templates or external dictionaries [28] [37] [42] [44] [45] [47], or sparsely annotated BSL-1K [3].Source: §1 · p. 2
- SMART uses MLLM-generated motion descriptions as semantic cues and SigLIP video-text alignment for stable training with small batches.Earlier CLIP-based CSLR methods used visual features with CTC supervision but lacked language-based semantic guidance.Source: §1 · pp. 2–3
- The Multi-Scale Temporal Adapter captures multi-scale temporal dependencies within the CLIP backbone during transformer encoding.It is designed to add temporal interactions while preserving pretrained CLIP representations.Source: §1 · p. 3
- CSFormer injects recognition-derived gloss evidence into boundary-aware spotting, while dense localization supervision complements weak CTC-based recognition.The framework uses complementary recognition and localization supervision to refine temporal alignments and support sign boundary estimation.Source: §1 · pp. 2–3, Figure 1 · p. 2
- Experiments cover PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL, using the two Korean datasets for CSLR and spotting for the first time.The reported results demonstrate complementarity between gloss-level recognition guidance and dense temporal supervision across different sign languages.Source: §1 · p. 3
2 Related Works
Prior work establishes CTC-based CSLR and dense temporal segmentation methods, but limited temporal supervision and underexplored dense sign spotting motivate SMART’s use of semantic cues and frame-level predictions.
Continuous Sign Language Recognition
- CTC-based CSLR uses sentence-level gloss supervision, while MLLM-generated descriptions remain underexplored as auxiliary semantic cues.Earlier systems used hand-crafted features or HMMs; deep-learning methods commonly combine spatial backbones with temporal modules, and AdaptSign [25] adapts a frozen CLIP backbone.Source: §2 · pp. 3–4
Peaky Alignment in CSLR
- CTC peaky alignments predict glosses as isolated spikes amid blanks, limiting temporal consistency; SMART integrates CSFormer for dense frame-level predictions.The cited work notes that sentence-level WER does not reveal this limitation and that sparse gradients to adjacent frames hinder modeling each sign’s temporal structure.Source: §2 · p. 4
Action segmentation and Sign Spotting
- Action segmentation advances include multi-stage temporal refinement in MS-TCN [12] and MS-TCN++ [34], attention-based ASFormer [48], boundary refinement in ASRF [26], and long-range modeling in LTContext.These approaches typically rely on pretrained visual features or optical flow, leaving the effectiveness of sign-domain backbones uninvestigated.Source: §2 · p. 4
- Prior sign spotting mainly uses short-query retrieval [28] [37] [42] [44] [45] [47] or sparse BSL-1K annotations [3], leaving dense frame-level recognition with large vocabularies largely unexplored.Sign spotting predicts temporal intervals for signs from a predefined vocabulary.Source: §2 · p. 4
3 Methods
SMART jointly trains sign-language recognition and spotting by combining MLLM-generated motion descriptions, temporally enhanced visual features, and recognition-guided boundary refinement.
3.1 MLLM Frame-Level Text Generation
- LLaVA-OneVision-7B [33] generates frame-level descriptions of signers’ hand movements and facial expressions, which serve as auxiliary semantic cues for CSLR video-text alignment, extending prior sign-description work [30].Per-frame text embeddings are averaged into one global video-level text embedding used for alignment during training.Source: §3.1 · p. 5, §3.2 · p. 7
3.2 Sign Language Recognition
- MSTA adds multi-scale temporal interactions to CLIP features, while SigLIP aligns video and text embeddings as auxiliary supervision during CSLR training.MSTA is inserted into CLIP’s last four transformer blocks to capture short- and long-range dependencies; SigLIP’s pairwise sigmoid loss supports stable alignment with small batches, and the alignment module is discarded at inference.Source: §3.2 · pp. 6–7
3.3 Sign Language Spotting
- CSFormer injects frozen recognition logits alongside visual features, exchanges information between independent streams through bidirectional cross-attention, and predicts gloss transitions with stage-wise boundary heads.At inference, temporally aligned recognition and spotting probabilities are linearly combined before CTC beam-search decoding.Source: §3.3 · pp. 7–8
4 Experiments
Experiments evaluate SMART on four sign-language benchmarks for recognition and spotting, finding strong performance across both tasks. Ablations and qualitative analyses examine its alignment, temporal-adaptation, and spotting components.
4.1 Experimental Settings
- Experiments cover four CSLR benchmarks, with spotting evaluated on the KSL datasets that provide temporal boundaries; recognition uses WER and spotting reports F1 at IoU 0.10, 0.25, and 0.50.Training uses CLIP ViT-B/16, MLLM-generated motion descriptions encoded by BERT, and MSTA in the last four transformer blocks; the spotting model is trained with the recognition module frozen.Source: §4.1 · pp. 8–10
4.2 Comparison with State-of-the-Art
- SMART achieves the best WER across all four benchmarks, with 96.72 F1@50 and 0.48 WER on Large-scale KSL and 59.77 F1@50 and 22.93 WER on DS KSL.The recognition comparison covers PHOENIX14-T, CSL-Daily, Large-scale KSL, and DS KSL; the spotting results are reported at IoU 0.50.Source: §4.2 · pp. 10–11, Table 2 · p. 10, Table 3 · p. 11
- CSFormer turns sparse recognition cues into dense spotting predictions, improving Test F1@50 from 7.37 to 96.72 on Large-scale KSL and from 5.90 to 59.77 on DS KSL.The spotting baseline fails to improve high-IoU localization on Large-scale KSL and underperforms CSLR-only across thresholds on DS KSL; action segmentation methods localize better but have higher WER.Source: §4.2 · pp. 10–11, Table 3 · p. 11
4.3 Ablation study
- On PHOENIX14-T, MSTA and alignment together achieve 17.58 Dev WER and 19.50 Test WER, while gated fusion of generated text features degrades recognition.MSTA reduces Dev WER from 18.60 to 17.88, and alignment alone reduces Test WER from 19.80 to 19.75.Source: §4.3 · pp. 11–12
- Adding CSLR features raises Test F1@50 from 94.50 to 95.44 on Large-scale KSL and from 58.54 to 59.68 on DS KSL; cross-attention reaches 97.06 and 59.81, respectively.The full CSFormer slightly sacrifices F1@50 relative to the best partial variant but achieves the lowest DS KSL Test WER, 22.93 versus 23.72.Source: §4.3 · p. 12
- On PHOENIX14-T, MSTA in the last four layers with r = 32 achieves 17.58 Dev WER; eight layers improve Test WER to 19.39 but worsen Dev WER to 17.88 and double adapter parameters.Under the same configuration, LoRA yields 24.40 Dev and 25.20 Test WER; batch size 8 is the practical upper bound, while SigLIP reaches 19.75 Test WER versus 20.76 for InfoNCE and 19.80 without alignment.Source: §4.3 · pp. 12–13
4.4 Qualitative Analysis
- Qualitative spotting results show SMART better matches ground-truth temporal boundaries and gloss labels than sparse CSLR-only predictions or dense ASFormer segments.SMART’s gloss predictions also correct examples missed by the baseline, including JETZT and ZEHNTE in one sequence and A and HOCH in another.Source: §4.4 · pp. 13–14, Figure 5 · p. 13
- Across four CSLR benchmarks, SMART attention focuses more on hand movements and facial expressions and less on background than the baseline.The attention visualizations compare dev and test samples.Source: §4.4 · p. 14, Figure 7 · p. 14
5 Conclusions
SMART unifies sign language recognition and spotting through MLLM-guided video-text alignment, temporal representation learning, and CSLR-guided localization. Experiments on four benchmarks show state-of-the-art performance and complementary benefits from recognition guidance and dense spotting supervision.
5 Conclusions
- SMART achieves state-of-the-art performance across four sign language benchmarks.The benchmarks cover sign languages, and additional spotting experiments show that CSLR representations provide gloss-level guidance while dense spotting supervision improves or preserves recognition depending on dataset characteristics.Source: §5 · p. 14
- SMART combines MLLM-generated motion descriptions for video-text alignment with a Multi-Scale Temporal Adapter for temporal representation learning.These components provide auxiliary semantic cues and model temporal interactions.Source: §5 · p. 14
- CSFormer injects recognition-derived gloss probabilities into a boundary-aware spotting module to refine temporal boundaries and produce dense frame-level sign predictions.The conclusion identifies gloss-level recognition guidance and dense temporal supervision as complementary for continuous sign language understanding.Source: §5 · p. 14