Source-linked AI summary
Investigating Temporal Motion Features for Pose-to-Text Indian Sign Language Translation
Manav Dhamecha, Praveen Kumar Chandaliya, Pruthwik Mishra
TL;DR
The study asks whether larger pretrained text-to-text transformers consistently benefit pose-to-text translation and whether explicit motion features are a more effective use of model capacity than scaling. Pose sequences are projected into pretrained T5 input embeddings, with a lightweight encoder and a T5-small variant that concatenates frame-to-frame pose differences with spatial features. T5-small + Motion raises BLEU from 0.189 to 0.298 (≈58% over spatial-only T5-small), while spatial-only T5-small leads BLEU, ROUGE-1, and ROUGE-L among spatial-only models, and T5-large leads chrF (15.59).
Problem
The study asks whether larger pretrained text-to-text transformers consistently benefit pose-to-text translation and whether explicit motion features are a more effective use of model capacity than scaling.
Source: §1 · p. 1Method
Pose sequences are projected into pretrained T5 input embeddings, with a lightweight encoder and a T5-small variant that concatenates frame-to-frame pose differences with spatial features.
Source: §1 · p. 1, §3.2 · p. 2Results
T5-small + Motion raises BLEU from 0.189 to 0.298 (≈58% over spatial-only T5-small), while spatial-only T5-small leads BLEU, ROUGE-1, and ROUGE-L among spatial-only models, and T5-large leads chrF (15.59).
Source: §5 · pp. 3–4Takeaways & Limitations
Under the evaluated configuration, enriching pose representations with explicit motion improves BLEU more than simply increasing T5 model size.
Source: §6 · p. 4The study uses one dataset and one training run per model, evaluates motion features only with T5-small, and describes the low-scoring systems as research baselines rather than deployable translators.
Source: §7 · pp. 4–5
Abstract
from arXiv · showhide
We investigate the effect of pretrained T5 model scale and explicit motion features on pose-to-text Indian Sign Language Translation (SLT) for the WSLP 2026 Shared Task. Pose sequences are projected into the embedding space of T5 through a lightweight pose encoder, with the complete model fine-tuned to generate English text. The shared task data used for this work consists of a test set with 5,334 examples and a validation set with 5,257 examples. We compare T5-small, T5-base, and T5-large, and additionally introduce a motion-augmented variant, T5-small + Motion, that adds explicit frame-to-frame pose differences to the input representation. T5-small achieves the best BLEU and ROUGE scores among the spatial-only models, while T5-large obtains the highest chrF score. Augmenting T5-small with motion features yields the largest single improvement observed in our study, substantially improving BLEU over the spatial-only baseline and making it the strongest model overall on this metric. Our submitted system ranked 5th on the official WSLP 2026 SLT testing leaderboard. The source code and trained models are publicly available on GitHub and HuggingFace.
1 Introduction
The study asks whether scaling pretrained T5 models or adding explicit motion information better supports pose-to-text Indian Sign Language translation. On the reported comparisons, motion augmentation substantially improves BLEU, while spatial-only model rankings vary by metric and training budget.
1 Introduction
- The work examines whether larger pretrained text-to-text transformers consistently help with non-textual pose inputs and whether explicit motion is more effective than scaling.Pose-based SLT represents frames with body and hand keypoints rather than raw video.Source: §1 · p. 1
- The study projects pose sequences into pretrained T5 input embeddings and fine-tunes T5-small, T5-base, T5-large, and T5-small + Motion on a shared held-out split.The motion variant concatenates frame-to-frame pose differences with spatial features; the best submission ranked fifth on the official WSLP 2026 testing leaderboard.Source: §1 · p. 1
- Among spatial-only scale variants, T5-small achieves the best BLEU, ROUGE-1, ROUGE-2, and ROUGE-L scores.T5-large achieves the best chrF score, while scaling does not produce monotonic improvement under the reported setup.Source: §1 · p. 2
- T5-small + Motion raises BLEU to 0.298 from 0.189 for spatial-only T5-small, an approximately 58% relative improvement.The motion-augmented model also improves chrF by approximately 2.0%.Source: §1 · p. 2
2 Related Work
Prior sign-language translation work moved from recurrent sequence-to-sequence systems toward transformers and reuse of pretrained language models. This work builds on pose-based SLT and studies T5 scale and explicit motion features for Indian Sign Language.
2 Related Work
- Neural SLT has shifted from recurrent sequence-to-sequence systems toward transformer-based architectures, including sign- or gloss-to-text translation (Camgoz et al., 2018, 2020; Yin and Read, 2020).Pretrained language models have also been reused by projecting non-textual or visual features into their representation space.Source: §2 · p. 2
- Moryossef et al. (2021) studied pose estimation for sign-language processing and introduced tools for pose representations.Zelezny and colleagues (2025) presented pose-based SLT with ablation studies and attention analysis using a T5-style encoder–decoder model.Source: §2 · p. 2
- This work adapts pose-to-text translation to Indian Sign Language using iSign and examines T5 model scale and explicit motion features derived from pose sequences.Source: §2 · p. 2
3 Methodology
The method converts pose files into frame-wise landmark features, optionally augments them with temporal differences, and maps the resulting sequences into T5 for text generation. A lightweight pose encoder and jointly fine-tuned T5 produce the translation.
3.1 Pose Representation
- Pose files are converted into frame-wise body and hand landmarks, represented by 300-dimensional spatial feature vectors with 3D coordinates and confidence values.Sequences are normalized per example and truncated or padded to 500 frames.Source: §3.1 · p. 2
3.2 Motion Features
- The motion variant computes first-order differences between consecutive spatial vectors and concatenates them with pose, producing a 600-dimensional per-frame input.The first frame's motion feature is set to zero; the concatenation supplies frame-wise velocity information alongside absolute pose without additional annotation.Source: §3.2 · p. 2
3.3 Pose-Conditioned T5
- A lightweight multilayer perceptron maps each per-frame feature vector to the corresponding T5 hidden dimension before the sequence enters T5 through its input embeddings.An attention mask excludes padded frames, and the pose encoder and all T5 parameters are jointly fine-tuned using teacher-forced text generation.Source: §3.3 · p. 2, Figure 1 · p. 3
4 Experimental Setup
The study uses official WSLP shared-task splits for leaderboard evaluation and a separate fixed iSign holdout for controlled model comparisons. Training and generation settings are specified for the four model variants.
4 Experimental Setup
- The official WSLP test and validation splits contain 5,334 and 5,257 pose-file examples, respectively, and leaderboard ranking uses test-set scores.The remaining released data is used for training.Source: §4 · p. 3
- All four models are compared on the same held-out 10% iSign split of 1,887 examples, distinct from the official WSLP test set and not used for training.This internal split supports comparison under identical conditions with an unlimited evaluation budget.Source: §4 · p. 3
- T5-small, T5-base, and T5-small + Motion train for 10 epochs, while T5-large trains for 3 because of higher computational cost.All use AdamW, batch size 16, gradient accumulation of 2, gradient clipping at 1.0, fp16, and a learning rate of 5 × 10^-4.Source: §4 · p. 3
- BLEU, chrF, and ROUGE are reported on the iSign holdout, using beam search with beam size 4 and a maximum generation length of 128 tokens.Table 1 reports results on all 1,887 held-out examples; ROUGE values are shown as percentages.Source: §4 · p. 3, Table 1 · p. 3
5 Results and Discussion
On the 1,887-example iSign holdout, adding motion to T5-small yields the strongest BLEU result, while spatial-only models show no consistent gain from scaling. T5-large leads chrF, but its shorter training and the scratch-trained pose encoder qualify interpretation of the comparisons.
5 Results and Discussion
- Among spatial-only models, T5-small leads four of five main measures, including BLEU at 0.189 and the strongest ROUGE-1 and ROUGE-L, while T5-large leads chrF at 15.59.The final comparison covers all 1,887 examples of the iSign 10% held-out split.Source: §5 · p. 3
- Spatial-only results do not improve monotonically with model size, but T5-large received three training epochs versus ten for T5-small and T5-base, and the intended warmup was not applied.The constant learning rate was 5×10−4; these conditions may disproportionately affect larger pretrained models.Source: §5 · pp. 3–4
- BLEU rises from 0.189 to 0.298 with T5-small + Motion, the largest improvement in the study, while T5-large still has the highest chrF.Motion also raises chrF from 14.44 to 14.73 and ROUGE-2 from 0.60 to 0.77; ROUGE-1 and ROUGE-L decrease slightly versus spatial-only T5-small.Source: Figure 2 · p. 4, §5 · p. 4
- The low absolute scores indicate that the challenge extends beyond scaling: the pose encoder is trained from scratch to map continuous pose sequences into T5’s embedding space.Source: §5 · p. 4
6 Conclusion
The paper compares three T5 scales and a motion-augmented T5-small for pose-to-text Indian Sign Language translation. In the evaluated setup, motion features improve BLEU substantially, whereas scale comparisons are constrained by unequal training conditions.
6 Conclusion
- The comparison covers T5-small, T5-base, T5-large, and T5-small + Motion for pose-to-text Indian Sign Language translation.Source: §6 · p. 4
- T5-small leads BLEU and ROUGE among spatial-only models, T5-large leads chrF, and adding frame-to-frame motion to T5-small substantially improves BLEU.Source: §6 · p. 4
- Because T5-large had fewer epochs and lacked the intended learning-rate warmup, the scale comparison reflects evaluated training configurations rather than a definitive capacity comparison.The authors propose matched budgets, improved optimization schedules, and testing motion features on larger T5 variants.Source: §6 · p. 4
7 Limitations
The experiments are limited to one dataset and one training run per model, with unequal training budgets and no multi-seed variance or human evaluation. Motion features were tested only with T5-small, and low absolute automatic-metric scores make the systems research baselines rather than deployable translators.
7 Limitations
- Experiments use one dataset and one training run per model, with unequal training budgets and no reported multi-seed variance or human evaluation.Source: §7 · p. 4
- The first-order motion difference was evaluated only with T5-small, so whether its improvement transfers to T5-base or T5-large remains unknown.Source: §7 · p. 5
- Low absolute automatic-metric scores mean the resulting systems should be treated as research baselines, not deployable translation systems.Source: §7 · p. 5
Ethics Statement
The work uses pose data provided for the WSLP 2026 SLT shared task and the public iSign corpus; it does not collect new participant data. Because incorrect translations can cause misunderstanding, the systems should not be used in accessibility-critical settings without substantial further validation.
Ethics Statement
- The study builds on provided WSLP 2026 pose data and the public iSign corpus, processed with an openly available pose-format library.Source: Ethics Statement · p. 5
- The study examines pose representations rather than raw video and collects no new participant data.Source: Ethics Statement · p. 5
- Without substantial further validation, the models should not be used in accessibility-critical settings because incorrect translations can cause misunderstanding.Source: Ethics Statement · p. 5