PaperlayerView exact revision on arXiv

Source-linked AI summary

OneSign: Unifying Sign Language Understanding Tasks with One Model

Shiwei Gan, Yafeng Yin, Xiao Liu, Desibieer Tuerdaken, Lei Xie, Sanglu Lu

arXiv:2609.33090v1cs.CLcs.AIcs.CV

TL;DR

SLU tasks are commonly handled with separate architectures and training pipelines, and existing unified approaches still use task- or dataset-specific finetuning; uniform decoder layers can also overlook sign–text modality differences. OneSign unifies ISLR, CSLR, and SLT in one framework and introduces modality-adaptive experts. Experiments across seven public benchmarks report strong performance across the tasks and datasets.

  • Problem

    SLU methods commonly use separate task-specific pipelines, while unified models still finetune by dataset and may process continuous sign representations and discrete text tokens uniformly.

    Source: §1 · p. 2

  • Method

    OneSign jointly trains ISLR, CSLR, and SLT within one framework and uses shared and modality-specific experts to model sign and text representations.

    Source: §1 · p. 2

  • Results

    OneSign achieves strong performance across seven public benchmarks spanning ISLR, CSLR, and SLT.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The paper concludes that one model checkpoint can support joint training and inference across the three SLU tasks without task-specific finetuning.

    Source: §6 · p. 9

    Joint training across heterogeneous datasets may dilute a dataset’s original distribution and adversely affect its corresponding test-set performance.

    Source: B. Limitations and Discussion · p. 15

Abstract

from arXiv · show

SLU encompasses a diverse set of tasks, including ISLR, CSLR, and SLT. Although these tasks share basic semantic and linguistic foundations, they are typically addressed with task-specific architectures and training pipelines, which hinders knowledge sharing and requires costly pretraining and finetuning for each task. In this paper, we focus on two aspects of SLU tasks: (1) training and inference pipelines are highly fragmented: most methods rely on pretraining on large-scale SL datasets followed by task- or dataset-specific finetuning, which leads to multiple specialized models rather than a single checkpoint. (2) current LLM-based methods may overlook the inherent modality discrepancy between sign and text tokens, simply concatenating them and processing both modalities with the same decoder layers. In this paper, we present OneSign, a unified framework that addresses multiple SLU tasks within a single model and a single checkpoint. OneSign reformulates ISLR, CSLR, and SLT under a single training paradigm. To accommodate the heterogeneous characteristics of sign and text representations, we introduce a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture, consisting of a shared expert and modality-specific experts for sign and text tokens. A modality router dynamically activates the corresponding experts, and their outputs are aggregated to form the final token representations. By enabling modality-dependent expert specialization while preserving a shared expert path, MA-MoE can effectively model the modality differences between continuous sign representations and discrete text tokens. Extensive experiments on multiple benchmarks demonstrate that OneSign achieves competitive or state-of-the-art performance on several benchmarks, highlighting its effectiveness as a unified SLU model. Datasets are available at : https://github.com/gswycf/OneSign.

1. Introduction

ISLR, CSLR, and SLT are closely related but commonly use separate architectures and pipelines, while prior unified models still require dataset-specific finetuning. OneSign combines the tasks in one framework and uses modality-adaptive experts to model sign and text differences.

1. Introduction

  • Prior unified SLU models still rely on large annotated datasets for pretraining and separate downstream finetuning, while processing sign and text tokens through the same decoder layers can overlook their modality gap.Sign representations are continuous, whereas text tokens are discrete; this difference may hinder cross-modal reasoning and representation learning.Source: §1 · p. 2
  • OneSign jointly trains and performs inference across ISLR, CSLR, and SLT with a single checkpoint, without task-specific pretraining or retraining.The paper reports experiments across multiple benchmarks and tasks.Source: §1 · pp. 2–3
  • MA-MoE combines an always-active shared expert with sign- and text-specific experts to model modality heterogeneity while preserving shared semantic modeling.A modality router dynamically activates the experts.Source: §1 · pp. 2–3
  • OneSign’s stated contributions include CSL-OpenWorld, a large-scale Chinese sign-language dataset with over 100k manually annotated video samples from diverse web sources and domains.The listed domains include news, law, and culture.Source: §1 · p. 3

2. Related Work

Prior work generally treats ISLR, CSLR, and SLT as separate tasks, with distinct supervision and modeling pipelines. Multi-task approaches and modality experts offer alternatives, but existing unified models still require dataset-specific finetuning, while OneSign uses a consistent modality-expert architecture at training and inference.

Related work on SLR

  • ISLR recognizes a single sign in a trimmed clip as a classification task, whereas CSLR recognizes unsegmented sequences and must align video inputs with gloss sequences.CSLR commonly uses CTC-based frameworks for weakly supervised temporal alignment.Source: §2 · p. 3

Related work on SLT

  • Gloss-based SLT commonly uses a pretrained CSLR backbone to provide gloss-aligned representations, while gloss-free SLT translates directly and uses textual alignment signals to address the lack of gloss supervision.Reported strategies include contrastive learning, pseudo-gloss objectives, text-aligned vector quantization, and multi-stage or curriculum-based pretraining.Source: §2 · p. 3

Unified SLU methods

  • Recent multi-task approaches jointly pretrain components on large sign-language datasets but still finetune separately for each task or dataset.The passage names UniSign (Li et al., 2025), MSLU (Zhou et al., 2025), and ShuBERT (Gueuwou et al., 2025b).Source: §2 · p. 3

Mixture of Experts

  • MA-MoE uses a shared expert and sign- and text-specific experts, and unlike VLMo and BEiT-3, keeps a consistent architecture during training and inference.The passage describes VLMo and BEiT-3 as requiring task-specific inference-structure adaptations.Source: §2 · p. 4

3. Method

OneSign brings sign-language recognition and translation tasks into a unified framework, combining pose-based modeling, modality-adaptive experts, and joint recognition–translation training. Its method also introduces CSL-OpenWorld, a diverse web-sourced dataset for open-domain sign-language translation.

3.1. Datasets

  • CSL-OpenWorld provides a web-sourced open-domain SLT dataset with over 12K tokens and coverage of varied topics, viewpoints, and nonstudio environments.It contains 100,458 videos, including 86,387 training samples, and is described as substantially larger than CSL-Daily and Phoenix14t.Source: §3.1 · p. 4

3.2. Preliminaries.

  • OneSign represents ISLR, CSLR, and SLT within a unified probabilistic framework trained by a single maximum-likelihood objective.The task cases specify a single sign label for ISLR, gloss sequences for CSLR, and gloss or pseudo-gloss sequences paired with spoken-language sentences for SLT.Source: §3.2 · p. 5

3.3. Model Architecture

  • The architecture uses pose-based visual features, a text encoder, and an MA-MoE decoder for unified sign–text modeling.Pose inputs are decomposed into torso, face, left-hand, and right-hand parts; Dynamic Part Graph Fusion models their spatial dependencies, while dataset-specific classifier heads share a visual backbone and are removed at inference.Source: §3.3 · pp. 5–6

3.4. Modality-Adaptive LLM MoE

  • MA-MoE combines an always-active shared expert with sign- and text-specific experts, routing tokens by modality to model modality-specific distributions while preserving shared semantics.A modality-specific load-balancing loss encourages even expert use while allowing specialization.Source: §3.4 · p. 6

3.5. Training

  • Training combines recognition and translation objectives, first warming up the visual backbone for 20 epochs before activating translation for 30 more.The recognition objective uses CTC supervision with multi-head classification, and the translation objective includes mixture-of-experts balancing losses.Source: §3.5 · p. 7

4. Ablation Study

The ablations examine multi-dataset classification, heterogeneous training, modality-specific experts, sparse expert configurations, dynamic part fusion, and runtime. Their findings favor dataset-specific classifier heads and modality-aware, dynamically weighted components, while identifying a selected sparse configuration and reporting training and inference speed.

A1: Effect of the Multi-Head Classification Layer

  • A dedicated classification head for each dataset improves performance and training stability over a shared head for multi-dataset training.Directly sharing a head dilutes the original dataset distributions and yields limited gains.Source: §4 · p. 7

A3: Effect of the Proposed MA-MoE

  • The full MA-MoE configuration with shared, sign-specific, and text-specific experts performs best across all reported metrics.Adding either modality-specific expert improves on using only the shared expert.Source: Table 4 · p. 7, §4 · p. 7

A4: Effect of Sparse MA-MoE Configurations

  • OneSign selects Ns = Nt = 2 with Top-k = 1 to balance performance and efficiency.Activating multiple experts can aggregate complementary information but requires more computation; activating one expert while increasing the total expert count yields minor gains.Source: Table 5 · p. 7, §4 · p. 7

A5: Effect of Dynamic Part Graph Fusion

  • WER falls from 21.1 to 20.1 and BLEU4 rises from 25.63 to 26.45 with Dynamic Part Graph Fusion.Fixed fusion graphs give modest, consistent gains over simple linear fusion, while dynamic fusion achieves the best performance on all reported metrics.Source: Table 6 · p. 7, §4 · p. 8

A2: Effect of Heterogeneous Training Datasets

  • Adding heterogeneous datasets without task-specific finetuning yields only modest gains, while OneSign still leverages complementary supervision.The reported evaluation is on Phoenix14t, and the passage notes that absent downstream finetuning can dilute original dataset distributions.Source: §4 · p. 7

A6: Training and Inference Speed

  • On Phoenix14t with one NVIDIA RTX 4090 GPU, training takes about 10–13 minutes per epoch and inference reaches 10–12 samples per second.Inference speed is measured on test samples averaging 250 frames over 100 runs.Source: Table 7 · p. 8, §4 · p. 8

5. Comparisons

OneSign is compared with task-specific methods across ISLR, CSLR, and SLT, including a jointly trained setting evaluated without further finetuning. It reports strong results across these tasks using pose inputs, with gains from joint training on the reported SLT benchmarks.

Evaluation on CSLR Tasks

  • On CSL-Daily and Phoenix14t, joint training without finetuning lowers WER to 25.7 and 20.1, respectively.Dataset-specific OneSign achieves 29.1 WER on CSL-Daily and 22.1 on Phoenix14t using pose inputs; the joint setting is comparable to or better than prior pose-based methods under a unified evaluation.Source: Table 9 · p. 8, §5 · p. 8

Evaluation settings

  • The joint setting trains on available dataset training splits and evaluates each benchmark without further finetuning; direct comparison with RGB-based models is not entirely fair because RGB inputs contain richer visual information.The paper contrasts this setting with training and evaluation on a single dataset.Source: §5 · p. 8

Evaluation on ISLR Tasks

  • On ASL Citizen, dataset-specific OneSign reaches 65.41 Top@1 versus 59.52 for ST-GCN, and joint training further improves performance across the reported ISLR benchmarks.The joint setting establishes a new baseline under the paper's unified protocol.Source: §5 · p. 8

Evaluation on SLT

  • On Phoenix14t and CSL-Daily, joint training raises BLEU4 from 26.22 to 26.45 and from 24.42 to 26.15, respectively.On How2Sign and CSL-OpenWorld, the jointly trained model without finetuning achieves the best or highly competitive results among pose-based methods on many metrics.Source: Table 10 · p. 9, Table 11 · p. 9, §5 · p. 9

6. Conclusion

OneSign unifies ISLR, CSLR, and SLT under one model and checkpoint, with joint training and inference without task-specific finetuning. It combines multi-head classification for heterogeneous datasets with modality-adaptive experts, and reports strong performance across seven public benchmarks.

6. Conclusion

  • OneSign integrates ISLR, CSLR, and SLT in a single model checkpoint, enabling joint training and inference without task-specific finetuning.The conclusion describes this as a unified paradigm for the three sign language tasks.Source: §6 · p. 9
  • Multi-Head Classifier Layers address vocabulary conflicts, while MA-MoE combines shared and modality-specific experts activated by a modality-aware router.The shared backbone is jointly optimized across heterogeneous datasets.Source: §6 · p. 9
  • Experiments across seven public benchmarks in ISLR, CSLR, and SLT show consistently strong performance across tasks and datasets.Source: §6 · p. 9

A. Impact Statement

The paper identifies potential accessibility benefits and research value, while describing its dataset as derived from public resources and used for research. It reports no personally identifiable information intentionally collected and no human-subject interaction or user-facing deployment.

A. Impact Statement

  • Potential positive impacts include improved information and communication accessibility for hard-of-hearing communities and support for future SL understanding research.Source: A. Impact Statement · p. 15
  • The dataset uses publicly accessible content for research, with no personally identifiable information intentionally collected and no human-subject interaction or user-facing deployment.Source: A. Impact Statement · p. 15
  • The authors foresee no significant negative ethical or societal consequences directly beyond those commonly associated with machine learning research using publicly available data.Source: A. Impact Statement · p. 15

B. Limitations and Discussion

The discussion identifies two constraints: pose-based inputs are chosen for efficiency despite possible pose-estimation noise, and joint training may dilute dataset-specific distributions. It leaves balancing multi-task and multi-dataset training as future work.

B. Limitations and Discussion

  • Joint training across heterogeneous datasets may dilute a dataset’s original distribution and adversely affect performance on its test set.The authors identify preserving dataset-specific characteristics while balancing multi-task and multi-dataset aggregation as future work.Source: B. Limitations and Discussion · p. 15
  • Pose data is used instead of raw RGB to balance training efficiency and performance under limited GPU resources, though DWPose noise may still affect performance.Prior pose-based SL models are described as effective and robust across multiple benchmarks, with pose-estimation errors not substantially hindering overall performance.Source: B. Limitations and Discussion · p. 15

C. Experiment

The experiments use seven public datasets spanning isolated recognition, continuous recognition, and gloss-based and gloss-free translation. The setup uses pose features, a unified model and training paradigm, and task-specific evaluation metrics.

C. Experiment

  • The ISLR benchmarks include WLASL2000, ASL Citizen, and Multi-VSL1000, covering word-level ASL, real-world crowdsourced signs, and multi-view Vietnamese signs.WLASL2000 contains 21,083 clips across 2,000 glosses; ASL Citizen contains approximately 84,000 clips of 2,731 signs; Multi-VSL1000 covers 1,000 categories.Source: C. Experiment · pp. 15–16
  • CSLR and gloss-based SLT use CSL-Daily and PHOENIX14T, while gloss-free SLT uses How2Sign and CSL-OpenWorld.CSL-OpenWorld is the proposed dataset, with 86,386 training, 7,033 validation, and 7,038 test samples.Source: C. Experiment · pp. 15–16
  • MSASL and OpenASL are excluded: many MSASL YouTube links were inaccessible, and OpenASL’s multiple signers per video do not suit the pose-based modeling.The authors state that further processing is needed before incorporating OpenASL results.Source: C. Experiment · p. 16
  • The authors report no sample- or video-level overlap among the independently collected datasets, so joint training does not introduce leakage through duplicated videos or samples.Training used four RTX 4090 GPUs with half precision and no additional external sign-language datasets beyond those described.Source: C. Experiment · p. 16
  • The model uses DWPose-extracted keypoints, a LLaMA 3.2 (1B) MoE backbone, and dataset and task descriptors to condition unified training.The default MoE setting uses two pose experts and two text experts, activating one expert in each modality; pseudo-glosses are derived from transcripts by punctuation removal, lemmatization, and word-level tokenization.Source: C. Experiment · p. 16
  • Evaluation follows dataset protocols, reporting Top-1/Top-5 accuracy for ASL Citizen and MultiVSL1000, P-I/P-C Top-1 for WLASL2000, WER for CSLR, and ROUGE-L F1 and BLEU-1/2/3/4 for SLT.The authors describe these metrics as enabling direct and fair comparison with prior methods.Source: C. Experiment · p. 16

D. More Ablation Study

The ablations examine conditioning descriptors, expert balancing and design, backbone variation, and activated expert counts. They report a best tested load-balancing coefficient of 0.5 and consistent MA-MoE gains across evaluated backbones.

Effect of the Dataset and Task descriptor

  • Using both dataset and task descriptors gives the best overall performance, while removing either one causes only marginal degradation.The descriptors condition the model toward corresponding data distributions without creating separate branches; the framework remains effective without them.Source: D. More Ablation Study · p. 17

Hyperparameter Analysis of the Load Balancing Coefficient

  • λ = 0.5 yields the lowest WER (21.8) and highest BLEU-4 (26.45) in the load-balancing analysis.Performance deteriorates as λ rises beyond 0.5; the authors attribute this to stronger balancing limiting expert specialization and routing flexibility.Source: D. More Ablation Study · p. 17

Comparison of different expert designs

  • Under comparable parameter scales, Dense FFN performs worst, while the authors attribute gains to sparse modality-specific adaptive routing rather than parameter scaling.The comparison includes shared plus modality-specific FFNs, Dense FFN, and sparse MoE without modality specialization.Source: D. More Ablation Study · pp. 17–18

Effect of MA-MoE on different LLM models

  • MA-MoE consistently improves performance with LLaMA3-1B, Gemma-3-1B, and Qwen-2.5-1.5B backbones.The authors characterize the approach as model-agnostic and integrable into varied LLM architectures.Source: D. More Ablation Study · p. 18, Table 16 · p. 18

Full ablation on the Impact of Activated Expert Count

  • The complete activated-expert-count ablation reports results on both DEV and TEST sets, extending the main paper’s DEV-only results.The supplementary results are identified as Table 17.Source: D. More Ablation Study · p. 18, Table 17 · p. 18

E. Qualitative Analysis

The qualitative analysis compares recognition and translation outputs across datasets, reporting more accurate and semantically coherent outputs with MA-MoE.

E. Qualitative Analysis

  • With MA-MoE, OneSign produces more accurate and semantically coherent recognition and translation outputs across the examined datasets.Recognition examples cover PHOENIX-2014T and CSL-Daily; translation examples cover those datasets plus How2Sign and CSL-OpenWorld.Source: E. Qualitative Analysis · p. 18

F. Dataset Visualization

The visualizations show that pose representations vary across seven datasets and illustrate the diversity and real-world complexity of CSL-OpenWorld.

F. Dataset Visualization

Related papers

Loading 2609.33090v1…