PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation

Keren Artiaga, Sabyasachi Kamila, Haithem Afli, Conor Lynch, Mohammed Hasanuzzaman

arXiv:2609.07965v1cs.CLcs.AI

TL;DR

SLT evaluation often overlaps signers across train and test, leaving uncertain whether models generalise to unseen individuals. The paper applies signer-fold cross-validation to three gloss-free models and finds lower performance under signer-independent evaluation, motivating more transparent benchmarking.

  • Problem

    Signer-overlapping SLT evaluation leaves uncertain whether models generalise to unseen signers rather than exploit signer-specific patterns.

  • Method

    The study applies signer-fold cross-validation to GFSLT-VLP, GASLT, and SignCL on PHOENIX14T and CSL-Daily.

  • Results

    Signer-independent evaluation lowers average translation performance compared with default-split baselines across the evaluated datasets and models.

  • Takeaways & Limitations

    The findings support signer-independent protocols as a way to expose generalisation gaps hidden by signer-dependent evaluation.

  • Takeaways & Limitations

    The evaluation was limited to models with publicly available implementations, excluding potentially stronger approaches without accessible code or pretrained models.

Abstract

from arXiv · show

Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev/test. This raises concerns about whether models truly generalise or instead rely on signer-specific regularities. We conduct signer-fold cross-validation on GFSLT-VLP, GASLT, and SignCL, three leading, publicly available, gloss-free SLT models, on CSL-Daily and PHOENIX14T. Under signer-independent evaluation, performance drops sharply: on PHOENIX14T, GFSLT-VLP falls from BLEU-4 21.44 to 3.59 and ROUGE-L 42.49 to 11.89; GASLT from 15.74 to 8.26; and SignCL from 22.74 to 3.66. We also observe that in CSL-Daily many target sentences are performed by multiple signers, so common splits can place identical sentences in both training and test, inflating absolute scores by rewarding recall of recurring sentences rather than genuine generalisation. These findings indicate that signer-dependent evaluation can substantially overestimate SLT capability. We recommend: (1) adopting signer-independent protocols to ensure generalisation to unseen signers; (2) restructuring datasets to include explicit signer-independent, sentence-disjoint splits for consistent benchmarking; and (3) reporting both signer-dependent and signer-independent results together with train-test sentence overlap to improve transparency and comparability.

1 Introduction

SLT evaluations commonly reuse signers across training and testing, leaving uncertain whether models generalise to unseen individuals. This study applies signer-fold cross-validation to quantify that gap and advocates signer-independent evaluation.

  • Overlapping signers across train, development, and test can let models exploit signer-specific patterns instead of learning to generalise.
  • The study applies signer-fold cross-validation to Phoenix14T and CSL-Daily across GFSLT-VLP, GASLT, and SignCL.
  • Signer-independent evaluation lowers average translation performance compared with default-split baselines, obscuring models’ true generalisation ability.
  • The study presents a systematic analysis of signer-independent SLT performance and advocates protocols that ensure robust and fair evaluation.

2 Related Works

Prior SLT work has largely reported improvements on signer-overlapping splits, so robustness to unseen signers remains insufficiently assessed. This study addresses that gap with signer-fold evaluation of three publicly available gloss-free models.

  • Most gloss-based and gloss-free SLT studies continue using signer-overlapping splits that do not assess generalisation to unseen signers.
  • Signer-fold cross-validation had not previously been systematically applied in SLT to evaluate robustness across all signers.
  • The study evaluates GFSLT-VLP, GASLT, and SignCL, three strong publicly available gloss-free models, under signer-fold protocols.

3 Experiments and Results

The experiments reorganise PHOENIX14T and CSL-Daily into signer-independent folds and compare results with default splits. Performance drops substantially on unseen signers, while sentence overlap is minimal in PHOENIX14T.

  • Experimental setup: Signer-fold evaluation withholds one signer for testing, another for development, and the remaining signers for training.PHOENIX14T uses nine folds; CSL-Daily uses ten signer folds and reports per-fold and mean aggregate scores.
  • Evaluation metrics: Corpus-level BLEU-4 and ROUGE-L are used to evaluate translation quality across the models.The models retain their original implementations and evaluation packages for comparability with prior work.
  • Results on PHOENIX14T: GFSLT-VLP drops from 21.44 BLEU-4 and 42.49 ROUGE-L on the default split to 10.53 BLEU-4 and 26.14 ROUGE-L across signer folds.GASLT falls from 15.74 to 10.24 BLEU-4, while SignCL falls from 22.74 to 4.18 BLEU-4.
  • Sentence overlap: PHOENIX14T dev/test sentences overlap training by only approximately 1–3% per fold, making inflation from repeated targets minimal.
  • Per-signer variation: SignCL averages just 4.18 BLEU-4 across folds, while GASLT shows smaller fold variability and more consistent signer-agnostic behaviour.GFSLT-VLP ranges from 17.30 BLEU-4 in Fold 6 to 3.59 in Fold 8; GASLT reaches 10.19 versus GFSLT-VLP’s 10.02 in Fold 3.
  • Implications: Signer-independent protocols reveal robustness gaps that standard splits can hide.

Analysis on the representative fold

Qualitative analyses across representative, best-, and worst-performing folds show that signer-independent evaluation exposes hallucination, semantic drift, template overgeneration, and uneven grounding. CSL-Daily additionally shows sentence-repetition effects that can reward memorised outputs, while limited compute restricts SignCL coverage.

  • Representative fold: SignCL is often least faithful to the input, producing unrelated, contradictory, or templated weather content.In contrast, GFSLT-VLP and GASLT more often preserve the reference topic, though both still introduce omissions, generalisations, or hallucinations.
  • Representative fold: GFSLT-VLP and GASLT can reproduce fixed expressions accurately, but GASLT also shows semantic drift, omitted details, and plausible-sounding incorrect content.Its strengths are clearest on stereotyped introductory phrases and common weather terminology.
  • Worst-performing folds: GFSLT-VLP frequently defaults to fixed phrases and repeated fallback outputs on its weakest fold, replacing specific weather events with generic descriptions.These outputs indicate template overgeneration and low diversity under signer-independent conditions.
  • Worst-performing folds: On the poorest fold, SignCL collapses toward one memorised phrase, whereas GASLT produces more diverse but still temporally and semantically distorted translations.The contrast suggests different failure patterns rather than reliable signer-robust grounding.
  • CSL-Daily: In CSL-Daily, repeated target sentences can make exact reproduction appear successful when the sentence was seen with other signers, while other predictions diverge severely.Many sentences overlap across train and test, favouring sentence-template memorisation over genuine generalisation.
  • CSL-Daily: GASLT and SignCL have incomplete signer-fold coverage because limited compute restricts SignCL to Folds 1–4, while GASLT is evaluated on all 10 folds.The completed SignCL folds show a consistent pattern relative to GASLT, but additional folds would refine the conclusion.

Analysis on the weakest-performing fold

The weakest CSL-Daily fold exposes severe semantic drift in GASLT predictions under signer-independent evaluation, with generic or unrelated outputs replacing input-grounded translations.

  • Analysis on the weakest-performing fold: Tables 10 and 11 provide qualitative GASLT-reference comparisons for CSL-Daily’s selected folds, while Table 12 focuses on Fold 2.The tables frame comparisons across weaker, stronger, and lowest-scoring fold examples.
  • Analysis on the weakest-performing fold: Fold 2 is CSL-Daily’s lowest-scoring GASLT fold and contains predictions with little relevance to the input meaning.Examples replace descriptive or instructional sentences with vague or interrogative utterances.
  • Analysis on the weakest-performing fold: GASLT frequently defaults to short, generic questions or unrelated dialogue-like phrases across the broader test set.Examples include “Did you see it?” and “Where is the subway station?”.
  • Analysis on the weakest-performing fold: Signer shift can trigger fallback behavior and semantic disconnection when GASLT fails to ground outputs in input semantics.The weakest fold contrasts with folds where content alignment is preserved.

4 Conclusion

The study concludes that signer-dependent evaluation limits trustworthy assessment of SLT generalisation. It recommends signer-independent, sentence-disjoint benchmarking and reporting practices that address signer and sentence-level confounds.

  • 4 Conclusion: Signer-dependent protocols are a central limitation in current SLT evaluation.Across folds and datasets, unseen-signer evaluation often produces lower scores, with effects varying between PHOENIX14T and CSL-Daily.
  • 4 Conclusion: Signer-independent evaluation should become the default, with splits enforcing sentence-level disjointness.The paper suggests splitting unique sentences first and then mapping them to clips.
  • 4 Conclusion: Signer-agnostic representations and training strategies should reduce exploitation of repeated sentence templates.Suggested strategies include frequency-aware sampling or downweighting repeated targets.
  • 4 Conclusion: Addressing signer dependence and sentence duplication is presented as a route toward more generalisable and practically deployable SLT systems.

Limitations

The evaluation is broad but bounded by model availability, input representation choices, and task coverage. These constraints leave open how inaccessible models, alternative modalities, and gloss-to-text systems perform under signer-independent evaluation.

  • Limitations: The study includes only models with publicly available implementations or pretrained models.Potentially stronger gloss-free approaches could not be evaluated without accessible code or checkpoints.
  • Limitations: The study does not evaluate alternative input representations such as skeleton-based features.Such features may be more robust to signer variability but may still retain signer-specific information.
  • Limitations: The study does not investigate gloss-to-text translation tasks.Gloss-based models could help disentangle signer identity from linguistic content.
  • Limitations: Despite these limitations, the findings support signer-independent protocols and dataset restructuring for evaluating generalisation beyond specific individuals.

Appendix A. Statistical Tests on Signer-Specific Folds on Phoenix14T

The appendix uses paired and one-sample statistical tests to compare signer-independent performance across models and against the default signer-dependent baseline. The tests support significant differences involving SignCL and lower signer-independent scores across all models and metrics.

  • Appendix A. Statistical Tests on Signer-Specific Folds on Phoenix14T: Paired t-tests and Wilcoxon signed-rank tests compare signer-independent scores across nine signer-specific PHOENIX14T folds.One-sample tests compare signer-independent scores with the default signer-dependent baseline.
  • Appendix A. Statistical Tests on Signer-Specific Folds on Phoenix14T: SignCL significantly outperforms GFSLT-VLP and GASLT across all signer-independent metrics.Both parametric and nonparametric tests report p-values below 0.05.
  • Appendix A. Statistical Tests on Signer-Specific Folds on Phoenix14T: No significant difference is observed between GFSLT-VLP and GASLT in signer-independent performance.The appendix describes their generalisation capabilities as comparable under these tests.
  • Appendix A. Statistical Tests on Signer-Specific Folds on Phoenix14T: Paired comparisons identify statistically significant differences between SignCL and the other models in BLEU-4 and ROUGE-L.
  • Appendix A. Statistical Tests on Signer-Specific Folds on Phoenix14T: Signer-independent performance is significantly lower than the default signer-dependent baseline across all metrics and models.Both one-sample t-tests and Wilcoxon signed-rank tests support this result.

Phoenix14T sentence complexity

Across Phoenix14T folds, sentence-length distributions vary, but the reported results show no consistent relationship between sentence complexity and translation performance.

  • Sentence-length proportions vary across Phoenix14T folds without a consistent pattern correlating with translation performance.Length categories use dataset-wide quartiles, keeping sentence classifications consistent across folds.

CSL-Daily sentence complexity

In CSL-Daily, signer-fold sentence-length distributions vary, but they do not consistently explain GASLT performance differences. Other factors, including signer-specific properties and repeated sentences across signers, may be more relevant.

  • Sentence-length distributions vary across CSL-Daily signer folds, but GASLT scores show no consistent relationship with those distributions.Fold 4 has the highest extra-short proportion at 40.29% without the highest scores, while Fold 5 has the highest long-sentence proportion at 34.54%.
  • Sentence-length distribution is not a primary explanatory factor for GASLT performance differences under signer-independent evaluation.The text avoids extending this conclusion to SignCL because its fold-wise evaluation remains incomplete.
  • Signer-specific properties and repeated sentences across signers are identified as alternative factors affecting signer-independent evaluation.Sentence repetition can create sentence-level video–text duplication under signer-only splits.
Loading 2609.07965v1…