Source-linked AI summary
Uni-Sign: Toward Unified Sign Language Understanding at Scale
Zecheng Li, Wengang Zhou, Weichao Zhao, Kepeng Wu, Hezhen Hu, Houqiang Li
TL;DR
Existing SLU pre-training methods leave a gap between pre-training and downstream tasks, especially when textual information or sufficiently large paired datasets are unavailable. Uni-Sign combines large-scale generative pre-training on CSL-News with unified SLT-based fine-tuning and multimodal fusion strategies. It achieves state-of-the-art performance across multiple SLU tasks, while the authors identify benchmark coverage and computational resources as remaining limitations.
Problem
Existing methods lack joint textual modeling or are limited by the scale and transferability of paired data, leaving a gap between pre-training and downstream SLU tasks.
Method
Uni-Sign uses generative pre-training on the 1,985-hour CSL-News dataset, treats ISLR, CSLR, and SLT as one SLT task during fine-tuning, and adds PGF with score-aware sampling.
Results
Uni-Sign achieves state-of-the-art performance across multiple downstream SLU tasks and improves SLT by up to 1.25 BLEU4 on CSL-Daily when sampling probability reaches 50%.
Takeaways & Limitations
A shared language-modeling objective and unified fine-tuning paradigm support transfer of SLU capabilities across ISLR, CSLR, and SLT.
Takeaways & Limitations
The work lacks a finely annotated, open-domain, large-scale benchmark and fine-tunes all parameters, creating substantial computational-resource demands.
Abstract
from arXiv · showhide
Sign language pre-training has gained increasing attention for its ability to enhance performance across various sign language understanding (SLU) tasks. However, existing methods often suffer from a gap between pre-training and fine-tuning, leading to suboptimal results. To address this, we propose Uni-Sign, a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks through a large-scale generative pre-training strategy and a novel fine-tuning paradigm. First, we introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing 1,985 hours of video paired with textual annotations, which enables effective large-scale pre-training. Second, Uni-Sign unifies SLU tasks by treating downstream tasks as a single sign language translation (SLT) task during fine-tuning, ensuring seamless knowledge transfer between pre-training and fine-tuning. Furthermore, we incorporate a prior-guided fusion (PGF) module and a score-aware sampling strategy to efficiently fuse pose and RGB information, addressing keypoint inaccuracies and improving computational efficiency. Extensive experiments across multiple SLU benchmarks demonstrate that Uni-Sign achieves state-of-the-art performance across multiple downstream SLU tasks. Dataset and code are available at github.com/ZechengLi19/Uni-Sign.
1 INTRODUCTION
Existing SLU pre-training methods either emphasize visual representation learning without joint textual modeling or rely on limited paired data, leaving a gap with downstream tasks. Uni-Sign addresses this through large-scale generative pre-training, unified fine-tuning, and the 1,985-hour CSL-News dataset.
- Visual self-supervised methods improve SLU representations but lack joint modeling of textual information, creating a gap with downstream SLT.
- Paired video-gloss or video-text pre-training incorporates language supervision but is constrained by data scale or downstream transfer capability.
- Uni-Sign introduces generative pre-training on large-scale data to capture semantics embedded in sign language.
- CSL-News contains 1,985 hours of Chinese Sign Language videos with textual annotations, providing the foundation for large-scale pre-training.
- Uni-Sign treats downstream ISLR, CSLR, and SLT as a single SLT task during fine-tuning, while PGF and score-aware sampling address keypoint inaccuracies and computational efficiency.
2 RELATED WORKS
Related work spans distinct SLU tasks, task-specific and self-supervised methods, sequence-to-sequence language modeling, and dataset construction. Uni-Sign instead uses a unified framework and introduces CSL-News to address the limited scale of Chinese Sign Language resources.
- SIGN LANGUAGE UNDERSTANDING: SLU includes ISLR, CSLR, and SLT, covering sign movement classification, sign-to-gloss sequence alignment, and text-sequence generation.
- SIGN LANGUAGE UNDERSTANDING: Task-specific methods use tailored designs such as classification heads for ISLR and CTC-based temporal alignment for CSLR.
- UNIFIED FRAMEWORK: Uni-Sign avoids task-specific fine-tuning designs and reports consistent performance across diverse SLU tasks.
- PRE-TRAINING: Self-supervised approaches learn representations from massive unlabeled data, while other work uses sequence-to-sequence modeling and language modeling to unify tasks.
- DATASETS: Existing SLT datasets include resources for multiple sign languages and tasks, but Chinese Sign Language datasets remain relatively underexplored.
3 METHOD
Uni-Sign builds a unified SLU framework around large-scale generative pre-training, shared fine-tuning objectives, and multimodal pose-RGB fusion. Its method combines the CSL-News dataset, staged training, and modules designed to address keypoint inaccuracies and computational cost.
- Dataset: CSL-News provides 1,985 hours of Chinese Sign Language video with textual annotations for large-scale pre-training.
- Training framework: Training proceeds in three stages: pose-only pre-training, RGB-pose interaction continued pre-training, and downstream task fine-tuning.
- Pose representation: Pose processing groups 69 selected keypoints into hands, body, and face, encodes them temporally, and concatenates group features before language-model input.
- Unified pre-training: Generative pre-training projects sign features into the language model dimension and trains language modeling over textual tokens.
- Unified fine-tuning: ISLR, CSLR, and SLT are treated as one SLT task during fine-tuning, using action descriptions, gloss sequences, or translation text as targets.
- Multimodal fusion: PGF uses keypoint coordinates as spatial priors to fuse RGB and pose features, while score-aware sampling selects RGB frames with low-confidence hand keypoints.
4 EXPERIMENTS
Uni-Sign is evaluated across recognition, continuous recognition, and translation benchmarks using separate pose-only and RGB-pose settings. The reported experiments compare these configurations with prior methods across multiple datasets and task-specific metrics.
- Implementation: Uni-Sign is pretrained on CSL-News for CSL and YouTube-ASL for ASL, then fine-tuned separately on each downstream dataset.
- Datasets: Evaluation covers ISLR on WLASL and MSASL, CSLR on CSL-Daily, and SLT on CSL-Daily, How2Sign, and OpenASL.
- Metrics: ISLR uses per-instance and per-class Top-1 accuracy, CSLR uses word error rate, and SLT uses BLEU and ROUGE-L, with BLEURT also reported for English datasets.
- Experimental settings: Experiments report both RGB-pose and pose-only settings, with pose-only experiments bypassing Stage 2 training.
- ISLR results: 4.09% and 2.47% improvements in per-instance Top-1 accuracy are reported over MSLU and NLA-SLR on ISLR benchmarks, respectively.
CSL-Daily (CSLR) CSL-Daily (SLT)
Uni-Sign’s unified fine-tuning and large-scale pre-training improve performance across CSLR and SLT, while ablations expose trade-offs among data scale, sampling, and fusion choices.
- CSL-Daily (CSLR): Only 1.3% and 0.7% performance drops relative to TS-SLR on MSASL1000 and WLASL2000 show competitive CSLR results without CTC loss.The paper attributes the gap to TS-SLR’s more complex architecture and dense intermediate-state constraints.
- CSL-Daily (SLT): 14.02 and 4.75 BLEU4 improvements over prior gloss-free SOTA on CSL-Daily dev and test sets demonstrate strong SLT performance.Uni-Sign also exceeds some gloss-based models and outperforms C2RL by 9.93 BLEU4 on OpenASL.
- Fine-tuning paradigm: 28.2 versus 37.4 and 27.4 versus 36.4 show that unified fine-tuning outperforms the mainstream CSLR fine-tuning paradigm in the reported comparisons.Improper fine-tuning of language-model features also reduces ISLR performance by 6.91% for P-I and 8.01% for P-C.
- Pre-training data scale: Performance progressively improves as the quantity of CSL-News pre-training data increases, highlighting the value of large-scale pre-training.The experiment samples portions of CSL-News to assess data-scale effects across tasks.
- Score-aware sampling: A 50% sampling probability yields a maximum 1.25 BLEU4 gain on the CSL-Daily test set, but substantially increases time consumption.The authors select 10% as the default to balance performance and speed.
- Fusion module: Deformable attention outperforms cross-attention in CSLR and SLT, supporting keypoint coordinates as priors in RGB-pose fusion.The comparison evaluates replacements within the PGF module.
5 CONCLUSION AND FUTURE WORK
Uni-Sign combines large-scale generative pre-training with unified SLU fine-tuning, supported by the CSL-News dataset and RGB-pose fusion components. The paper reports notable improvements across SLU tasks and identifies broader scenarios for future study.
- Conclusion: CSL-News provides 1,985 hours of Chinese Sign Language video-text pairs for large-scale pre-training.The framework treats downstream SLU tasks as a single SLT task during fine-tuning.
- Conclusion: Uni-Sign narrows the pre-training–downstream gap by unifying fine-tuning and incorporating PGF and score-aware sampling for RGB-pose cues.These components target efficient visual-cue capture and a performance-speed trade-off.
- Future work: Future work includes multilingual SLU models, complex scenarios, and sign language production.The listed complex scenarios include complex backgrounds, multi-signer situations, and long-duration understanding.
A.1 FRAMEWORK IMPLEMENTATION
The implementation extracts whole-body pose groups, encodes them with separate modules, and uses a pre-trained language model. Uni-Sign is presented as parameter-efficient relative to compared methods.
- Framework implementation: Whole-body keypoints are divided into left hand, right hand, face, and body groups for pose representation.RTMPose-x extracts the keypoints, with selected indices assigned to each group.
- Framework implementation: Separate linear, pose, and temporal encoders are used for each group rather than sharing weights.This design captures each group’s representation individually.
- Framework implementation: Uni-Sign uses the HuggingFace pre-trained google/mt5-base language model.The implementation identifies the model source through HuggingFace.
- Framework implementation: Uni-Sign demonstrates advantages in parameter efficiency and performance compared with previous methods.The parameter comparison is reported in Table 12.
A.2 ADDITIONAL ABLATION STUDIES
Additional studies examine pose composition, score-aware sampling, visual examples, and translation behavior. The results support using all sub-poses while revealing failures on complex sentence structures.
- Additional ablation studies: Each evaluated sub-pose is indispensable, motivating inclusion of all sub-poses in Uni-Sign.The study uses training-from-scratch pose-only settings to reduce time consumption.
- Additional ablation studies: Score-aware sampling computes sampling scores from keypoint-confidence-based reliability and uses a sampling probability.The pseudocode defines pose features, confidence scores, coordinates, and P_samp.
- Additional ablation studies: Uni-Sign achieves notable CSLR performance on CSL-Daily, while failure cases expose difficulty distinguishing semantically similar signs.The accompanying visualization presents representative ISLR and CSLR examples.
- Additional ablation studies: Generated translations are usually close in meaning to references despite occasional structural differences, but complex sentence structures remain difficult.The examples span CSL-Daily, How2Sign, and OpenASL.
A.5 MORE DISCUSSION ABOUT CSL-NEWS DATASET
CSL-News combines long duration with diverse, standardized news-based signing and comprehensive CSL knowledge. Its design supports robust sign language understanding and generalization while using privacy-preserving inputs.
- Dataset characteristics: CSL-News covers diverse news topics rather than restricting data to specific domains.Its topics include culture, economy, sports, science, and daily life.
- Dataset characteristics: News segments featuring sign language experts provide more standardized signing than YouTube-scraped datasets.The paper links this standardization to dataset reliability and overall quality.
- Dataset impact: Models pretrained on CSL-News acquire robust sign language understanding and generalization capabilities.The paper attributes this capability to the dataset’s comprehensive CSL knowledge.
- Dataset analysis: Figure 8 presents the vocabulary distribution of the CSL-News dataset.The figure is provided as part of the dataset analysis.
- Input design: Uni-Sign uses keypoints and cropped hand video clips as inputs to protect Deaf/Hard of Hearing communities’ privacy.The paper presents privacy protection alongside performance as a benefit of this input design.
C LIMITATIONS
The paper identifies two limitations: the lack of a finely annotated, open-domain, large-scale benchmark and the computational burden of full-parameter fine-tuning.
- No finely annotated, open-domain, large-scale benchmark is available to fully investigate Uni-Sign’s capabilities and limitations.
- Fine-tuning all Uni-Sign parameters at every training stage creates urgent computational-resource challenges.
D COMPLETE RESULTS OF EXPERIMENTS
The complete experimental results are supplied outside the main paper because page limitations required omitting some results. The supplementary tables cover ISLR and SLT benchmarks across several datasets.
- Complete results: Some experimental results are omitted from the main paper because of page limitations.The complete results are provided to support future reference.
- ISLR: Table 18 reports ISLR results on various benchmarks, distinguishing reproduced methods and the best previous and Uni-Sign results.
- SLT: Tables 19–21 report SLT results on CSL-Daily, OpenASL, and How2Sign, with dataset-specific markers for reproduced methods and gloss-based results.