PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment

Keren Artiaga, Yang Li, Ercan Engin Kuruoglu, Wai Kin, Chan

arXiv:2608.16804v1cs.AI

TL;DR

Sign language recognition lacks resources across many languages, motivating transfer from higher-resource sign-language domains to low-resource ASL. The paper uses TA3N-based domain adaptation with multi-scale temporal alignment and finds that it outperforms pre-training, with shorter-term alignment consistently favored across settings and modalities.

  • Problem

    Sign Language Recognition lacks high-resource datasets across most of the world’s sign languages, limiting evidence for low-resource recognition, including ASL.

  • Method

    The paper applies TA3N-based domain adaptation with a Temporal Relational Network to align multi-scale temporal relations between LSA or CSL and ASL using RGB and Optical Flow inputs.

  • Results

    Multi-scale temporal domain adaptation between LSA or CSL and ASL performed significantly better than pre-training, while shorter-term TRNs 3, 5, or 7 outperformed longer-term TRNs 10 or 15 across settings and modalities.

  • Takeaways & Limitations

    Cross-sign-language domain adaptation improved SLR under low-resource conditions, including full-scale and few-shot settings across RGB and Optical Flow experiments.

  • Takeaways & Limitations

    The study uses a particularly challenging ASL dataset with diverse signers and recording conditions, and effective sign-language modeling also requires language modeling because sign and spoken languages differ grammatically.

Abstract

from arXiv · show

Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.

1 Introduction

The paper addresses limited high-resource data for sign language recognition through cross-sign language transfer learning and domain adaptation. It evaluates multi-scale temporal alignment, learning settings, and RGB versus Optical Flow inputs for transfer to ASL.

  • Sign Language Recognition is under-researched across the 135 sign languages, partly because most lack high-resource datasets.The paper identifies low-resource data as a challenge for recognition tasks.
  • Cross-sign language transfer learning transfers knowledge from related sign-language domains to learn high-level visual features unavailable from general non-sign language datasets.The proposed approach targets features specific to signing rather than only low-level features from generic datasets.
  • The study applies domain adaptation between LSA or CSL as source domains and ASL as the target domain.It examines transfer in both full-scale and few-shot settings.
  • Multi-scale temporal alignment compares short-term and long-term temporal relations between source and target sign-language domains.Temporal relations are groups of select time-ordered video frames.
  • The experiments compare RGB and Optical Flow modalities across the transfer-learning settings.Optical Flow was generated from video frames without specialized camera equipment and was implemented using the Gunner Farneback algorithm.

2 Related Works

Prior sign language transfer-learning studies largely rely on generic pre-training and often address static or limited-sign settings. This paper positions cross-sign-language domain adaptation as a response to domain shift and low-resource, diverse video data.

  • Neural network-based transfer learning: Earlier SLR transfer-learning studies commonly initialize models with weights from large-scale datasets such as ImageNet.Reported applications include VGG16, CNN, InceptionV3, and hybrid transfer-learning models.
  • Domain-specific transfer learning: Domain-specific transfer learning has also been explored, including pre-training between British Sign Language and American Sign Language.The cited BSL-to-ASL study used 18-class datasets and ASL signs from two subjects.
  • Research gaps: Existing studies often focus on static alphabet, character, or digit signs and use limited numbers of subjects.The paper contrasts these settings with its focus on dynamic word-level ASL signs involving one- and two-handed gestures.
  • Research gaps: Few-shot transfer learning had not previously been explored in SLR, while video-domain adaptation methods such as TA3N had mainly been tested on large-scale action-recognition datasets.The paper selects TA3N with its TRN module as the architecture for cross-sign-language domain adaptation.
  • Domain adaptation: Domain adaptation addresses domain shift when source and target datasets have different probability distributions despite sharing a task.It reduces this discrepancy by adjusting the target distribution toward the source distribution.

3 Technical Approach

The approach preprocesses sign-language videos into RGB and Optical Flow frames, then applies TA3N domain adaptation with multi-scale temporal relations. It compares temporal scales and transfer settings to identify how source–target alignment should be performed for cross-sign-language recognition.

  • 3.1 Pre-processing of Videos: The pipeline converts videos into RGB and Optical Flow frames before domain adaptation.The resulting frame features are used as inputs to the transfer-learning process.
  • 3.2 Domain Adaptation: TA3N aligns multi-scale temporal relation features from source and target sign-language domains using domain-adversarial training.The architecture extracts frame-level and task-driven features, forms N-frame relations, and performs class and domain prediction.
  • 3.2.1 Temporal Relations Network: The TRN module fuses features from sparsely sampled, time-ordered frames into temporal relations at multiple time scales.For example, a 2-frame relation combines two chronologically ordered frames, and the multiscale TRN accumulates relations with different N values.
  • 3.2.1 Temporal Relations Network: The study evaluates five multi-scale TRN settings: 3, 5, 7, 10, and 15 temporal relations.Values 3, 5, and 7 represent shorter-term relations, whereas 10 and 15 represent longer-term relations.
  • 3.2.1 Temporal Relations Network: Shorter-term temporal relations are evaluated as a proposed way to improve adaptation across sign languages from different families.The paper motivates shorter-term relations as potentially reducing overfitting when source and target sign languages differ substantially.
  • 3.3 Transfer Learning Settings: Experiments include full-scale and few-shot transfer learning, with the few-shot setting using limited target-domain training data.The few-shot setup uses a 20:80 ratio of target training data to target test data, while the full-scale setup uses 80:20.

4 Experiments

The experiments adapt sign-language recognition across domains and compare temporal-relation choices, learning settings, and input modalities. They also evaluate whether domain adaptation suits the low-resource characteristics of sign-language datasets.

  • TA3N required modification because its default implementation performed poorly on low-resource sign-language recognition compared with its original large-scale action-recognition setting.The original tests used HMDB51 and UCF101 with ResNet-101; this study adapted the method for sign-language recognition.
  • The study compares shorter-term and longer-term multiscale TRNs, full-scale and few-shot transfer learning, and RGB versus Optical Flow inputs.These comparisons structure the reported sign-language recognition experiments.

4.1 Experiment Setup

The setup uses overlapping sign classes from LSA, CSL, and ASL datasets, with experiments spanning different dataset sizes, model complexities, validation splits, and multiscale TRN counts. The target ASL data are especially limited relative to the source datasets.

  • Datasets and classes: The experiments use LSA64, Chinese Isolated Sign Language, and WLASL300 for LSA, CSL, and ASL, respectively, selecting labels shared between source and target datasets.The selected overlap includes 26 mutual LSA–ASL labels and 23 mutual CSL–ASL labels.
  • Dataset characteristics: The WLASL300 subset averages 1.3 gesture repetitions per subject, compared with 5 repetitions in LSA64 and Chinese Isolated SLR.This difference is presented as evidence of greater generalization ability in WLASL, while also raising an underfitting concern.
  • Dataset characteristics: WLASL300 has fewer than 20 samples per class, versus 50 in LSA64 and 250 in Chinese Isolated SLR.The target dataset therefore provides substantially fewer samples per class than either source dataset.
  • Evaluation and models: Tables 3 and 4 report learnable-parameter counts for LSA-to-ASL and CSL-to-ASL domain-adaptation models across N-multiscale TRNs.The tables describe model complexity separately for the two source-to-target adaptations.
  • Evaluation and models: Experiments use randomized 5-fold cross-validation, up to 100 training epochs, batch size 20, and 80:20 full-scale or 20:80 few-shot training-to-testing ratios.Both full-scale and few-shot domain adaptation compare 3, 5, 7, 10, and 15 multiscale TRNs and report mean accuracy with standard deviations.

4.2 Experiment Results and Analysis

Domain-adaptation results favor shorter-term multiscale temporal relations in full-scale and few-shot settings, while performance varies by source language, modality, and training regime. RGB generally produces stronger domain-adapted models, but some optical-flow configurations are competitive or preferable.

  • Full-scale LSA to ASL: 12.93% accuracy with a 5-multiscale TRN is the best RGB result for full-scale LSA-to-ASL adaptation, improving 59.62% over baseline.The 5-multiscale TRN is identified as the best RGB configuration for this group.
  • Full-scale LSA to ASL: For full-scale LSA-to-ASL Optical Flow, 3-multiscale TRN reaches 10.08% accuracy, while 7-multiscale TRN gives the largest baseline improvement at approximately 92.017%.The 7-multiscale model has 9.958% accuracy, the second-highest in that group.
  • Full-scale CSL to ASL: Full-scale CSL-to-ASL reaches 11.86% in RGB and 10.08% in Optical Flow with 7-multiscale TRN, improving over baseline by 24.19% and 10.31%, respectively.For RGB, 7-multiscale TRN is preferred; for Optical Flow, 3-multiscale TRN is preferred despite 7-multiscale reaching the highest accuracy.
  • Full-scale domain adaptation: Shorter-term multiscale TRNs such as 3, 5, and 7 generally outperform longer-term TRNs against strong baselines in full-scale domain adaptation.This pattern is reported across the full-scale experiments.
  • Few-shot LSA to ASL: In few-shot LSA-to-ASL adaptation, 3-multiscale TRN achieves 6.85% RGB accuracy, while 10-multiscale TRN achieves 7.34% Optical Flow accuracy and a 26.48% baseline improvement.The 3-multiscale RGB result improves 7.03% over its corresponding baseline.
  • Few-shot CSL to ASL: Few-shot CSL-to-ASL achieves 9.266% RGB and 7.948% Optical Flow accuracy with 7-multiscale TRN, improving over baseline by 3.41% and 4.55%.For Optical Flow, 3-multiscale TRN is considered most effective because it reaches 7.788% accuracy and a 12.54% improvement.
  • Overall comparison: Full-scale learning produces higher-accuracy domain-adapted models, and RGB is generally superior to Optical Flow across the reported experiments.LSA as the source reaches the highest accuracy in all domain-adaptation experiments, but underperforms CSL in few-shot settings because its smaller source domain provides fewer representations.

4.3 Pre-training

The study compares neural-network pre-training with domain adaptation for transferring sign-language knowledge to ASL. Pre-training yields improvements less often than domain adaptation under the evaluated full-scale and few-shot settings.

  • Pre-training method: Neural-network pre-training initializes target training with source-domain weights using a ResNet-50 architecture and an added Long short-term memory layer.The approach follows a source-to-target pre-training procedure rather than multiscale temporal domain adaptation.
  • Experimental setup: The pre-training experiments use 5-fold cross-validation, up to 100 epochs, batch size 2, and 80:20 full-scale or 20:80 few-shot training-to-testing ratios.These settings are applied to both full-scale and few-shot pre-training.
  • Results: Only LSA-pre-trained ASL in RGB shows positive transfer for full-scale learning, while only CSL-pre-trained ASL in Optical Flow shows positive transfer for few-shot learning.The comparison uses the ASL RGB and Optical Flow baselines in the reported tables and figures.
  • Results: Pre-training improves over baseline in 25% of cases, compared with 90% for domain adaptation.The domain-adaptation comparison counts 18 improvements among 20 N-multiscale TRNs, with two negative adaptations.
  • Variability: Few-shot pre-training test results are closer to their mean than full-scale pre-training results, consistent with the domain-adaptation experiments.The supplied passage characterizes this through the observed standard deviations.

5 Conclusion

The study reports that multi-scale temporal domain adaptation improves cross-sign-language recognition in low-resource settings, including when source and target languages differ. It also identifies shorter-term temporal alignment as more beneficial in the tested settings, while noting remaining challenges for optical-flow use and language-level translation.

  • The study examines cross-language adaptation from LSA and CSL to ASL in RGB and optical-flow modes and reports few-shot domain adaptation between sign languages.
  • Multi-scale temporal domain adaptation performed significantly better than pre-training ASL with LSA or CSL, which produced more negative than positive transfers.
  • Shorter-term multi-scale TRNs, such as 3, 5, or 7, benefited adaptation more than longer-term TRNs, such as 10 or 15, across full-scale and few-shot settings and both input modes.
  • RGB performed better than optical flow in the experiments, while future work is proposed to investigate optical flow’s further potential for sign-language recognition.
  • Effective sign-language modeling for translation requires language modeling because spoken and sign languages differ in grammar structure, although domain adaptation may address these differences.
  • Finding similar gloss annotations across datasets remains a significant challenge for adapting between sign-language datasets and producing transcriptions.

Declarations

The authors disclose no relevant financial or non-financial competing interests and identify the research funding sources. All authors contributed to the study and approved the final manuscript.

  • The authors report no relevant financial or non-financial interests to disclose.
  • The research was funded by multiple Shenzhen, Guangdong, and national science and technology programs listed by the authors.
  • All authors contributed to conception and design, analysis, manuscript development, revision, and approval of the final manuscript.
Loading 2608.16804v1…