Source-linked AI summary
Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment
Youngjoon Jang, Liliane Momeni, Zifan Jiang, Joon Son Chung, Gül Varol, Andrew Zisserman
TL;DR
The paper addresses the need for unified sign language translation and sign-subtitle alignment under multimodal, temporal, and annotation constraints. It combines privacy-preserving pose and lip features, a Sliding Perceiver, and joint LLM training, achieving state-of-the-art results across tasks with cross-linguistic transfer.
Problem
Sign language understanding must handle manual and non-manual cues while jointly supporting translation and temporally noisy subtitle alignment, whose annotations are costly and limited in scale.
Source: §1 · pp. 1–2Method
The model combines a lightweight pose-and-lip visual backbone, a Sliding Perceiver that forms word-level embeddings, and prompt-controlled joint LLM training for SLT and SSA.
Source: §1 · p. 2Results
Takeaways & Limitations
Combining pose and lip features enables lightweight end-to-end representations that transfer across sign language understanding tasks, languages, datasets, and signers.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Our aim is to develop a unified model for sign language understanding, that performs sign language translation (SLT) and sign-subtitle alignment (SSA). Together, these two tasks enable the conversion of continuous signing videos into spoken language text and also the temporal alignment of signing with subtitles -- both essential for practical communication, large-scale corpus construction, and educational applications. To achieve this, our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images while preserving signer privacy; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings to bridge the vision-text gap; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA, reinforcing both linguistic and temporal alignment. To promote cross-linguistic generalisation, we pretrain our model on large-scale sign-text corpora covering British Sign Language (BSL) and American Sign Language (ASL) from the BOBSL and YouTube-SL-25 datasets. With this multilingual pretraining and strong model design, we achieve state-of-the-art results on the challenging BOBSL (BSL) dataset for both SLT and SSA. Our model also demonstrates robust zero-shot generalisation and finetuned SLT performance on How2Sign (ASL), highlighting the potential of scalable translation across different sign languages.
1. Introduction
The paper develops a unified model for sign language translation and sign-subtitle alignment, addressing the multimodal, temporal, and data-scarcity challenges of sign understanding. Its lightweight visual backbone, Sliding Perceiver, and joint LLM training support scalable multilingual modeling across these tasks.
Motivation
- The model jointly performs sign language translation and sign-subtitle alignment, supporting communication, corpus construction, education, and sign-segment retrieval.SLT converts signing into spoken-language text, while SSA localizes signing relative to subtitles despite noisy temporal correspondence.Source: §1 · p. 1
Challenges
- Limited signer diversity, costly annotations, frozen visual backbones, and simple mapping networks constrain existing SLT and SSA systems.Manual subtitle alignment requires around 10–15 hours per hour of video, while major datasets contain only 9–39 signers.Source: §1 · p. 2
Contributions
- A lightweight backbone combines human pose keypoints with lip features to capture manual and non-manual cues while improving privacy, domain generality, and end-to-end finetuning.Keypoints reduce nuisance factors such as clothing and lighting, while lip features capture non-manual aspects of signing.Source: §1 · p. 2
- The Sliding Perceiver summarizes local feature windows into moving latent representations that produce word-level embeddings and better align visual sequences with linguistic units.It addresses the inability of frame-wise MLP mappings to model short gesture sequences corresponding to single words.Source: §1 · p. 2
- A single prompt-controlled LLM integrates SLT and SSA, and the combined design achieves state-of-the-art results across both tasks and British and American Sign Languages.The authors report state-of-the-art SLT on BOBSL and How2Sign and state-of-the-art SSA on BOBSL.Source: §1 · p. 2
2. Related Works
Related work spans pose and appearance-based recognition, gloss-based and gloss-free translation, and temporal alignment methods. The paper builds on efficient multimodal representations and LLM adaptation while introducing an LLM-based approach to scalable sign-subtitle alignment.
Isolated sign language recognition
- Pose-based recognition improves efficiency, signer anonymity, and robustness, while combining holistic keypoints with grayscale lip regions recovers manual and non-manual information.The proposed lightweight ISLR backbone outperforms Video Swin Transformer and supports downstream translation and alignment.Source: §2 · p. 2
Sign language translation
- Gloss-based SLT uses intermediate gloss predictions, whereas gloss-free SLT directly maps video to sentences and avoids costly gloss annotations for scalable open-vocabulary generalization.The paper adopts the gloss-free approach to exploit large-scale video-text pairs across signers and languages.Source: §2 · pp. 2–3
- Recent gloss-free systems adapt pretrained visual features and LLMs through visual tokens, finetuning, contrastive alignment, or parameter-efficient methods such as LoRA.This work finetunes Flan-T5 with visual representations while updating the visual backbone end to end.Source: §2 · p. 3
Sign–subtitle alignment
- Sign-subtitle alignment methods have used sparse correspondences, segmentation, linguistic knowledge, and broadcast data, but many operate at coarser temporal levels or depend on assumptions about subtitle order.The paper presents an LLM-based framework intended to make alignment more accurate and scalable.Source: §2 · p. 3
- Unlike prior joint SLT and caption-alignment work reporting 24.3% frame accuracy, this model uses both video and subtitles to localize signing segments and generate sign-aligned subtitles.Subtitle input complements visual evidence for practical alignment from audio-aligned transcripts.Source: §2 · p. 3
3. Method
The method unifies sign language translation and subtitle alignment with privacy-preserving visual streams, local temporal aggregation, and multitask training. It combines pose and lip features with an LLM through staged multilingual pretraining and task-specific objectives.
3.1. Architecture
- 3D keypoints are grouped into face, body, and hand articulators, encoded separately, jointly projected, and processed by a Conformer instead of average pooling.The architecture preserves detailed joint interactions through reshaping and learnable projection before temporal encoding.Source: §3.1 · pp. 3–4
- Lip-region frames provide non-manual cues through a frozen pretrained lip reader and a trainable Conformer, while the Sliding Perceiver aggregates local multimodal windows into latent representations for the LLM.The Perceiver uses cross-attention and feed-forward layers with a learnable latent query over sliding windows.Source: §3.1 · p. 4, Figure 4 · p. 4, Figure 3 · p. 4
3. Method
- The model combines pose and lip backbones with a Sliding Perceiver and Flan-T5-XL for prompted sign language translation and sign-subtitle alignment.The pose and lip backbones are pretrained independently before being combined with the mapping network and language model.Source: §3 · p. 3, §3.2 · p. 4, §3.1 · p. 4
3.2. Training overview
- Training uses a staged curriculum: mapping-only SLT pretraining, full-model SLT training, multitask SLT–SSA training, and target-dataset finetuning.Pretraining uses 2,700 hours from ASL and BSL data in YouTube-SL-25 and BOBSL, with multitask training retained during finetuning.Source: §3.2 · pp. 4–5
3.3. Visual backbone ISLR training losses
- The visual backbones use cross-entropy, articulator auxiliary, and contrastive losses, with contrastive alignment between visual features and pretrained-LLM text embeddings.Duplicate pseudo-gloss labels are reweighted by inverse batch frequency in the contrastive objective.Source: §3.3 · p. 5
3.4. SLT and SSA training losses
- SSA predicts six timestamp digits using categorical cross-entropy plus digit-wise L1 loss, combining exact token matching with numerically precise temporal alignment.Each digit distribution is soft-decoded into an expected numeric value before the L1 term is computed.Source: §3.4 · p. 5
4. Experiments
Experiments evaluate the model across datasets, implementation settings, ablations, recognition, translation, and alignment, with comparisons spanning BSL and ASL tasks.
4.1. Data and evaluation protocols
- The experiments use BOBSL, YouTube-SL-25, How2Sign, and FLEURS-ASL, evaluating ISLR, SLT, and SSA with task-specific metrics.BOBSL-SIGN supplies pseudo-gloss ISLR data, while BOBSL-SENT uses signing-aligned subtitles; YouTube-SL-25 contributes ASL and BSL pretraining data, and How2Sign and FLEURS-ASL support ASL evaluation.Source: §4.1 · pp. 5–6, Table 1 · p. 6
4.2. Implementation details
- The implementation combines pose and lip backbones with a Sliding Perceiver, large-scale pretraining, multitask sampling, and task-specific 20-second inference windows.Multitask training samples SLT and SSA at an 80%:20% ratio, while SSA uses dynamic time warping and timestamp decoding during inference.Source: §4.2 · pp. 6–7
4.3. Ablation study
- Removing visual-backbone components consistently degrades ISLR, while the Sliding Perceiver outperforms MLP, convolutional, and Token Shuffle mappings across SLT metrics.The ablations test contrastive and auxiliary losses, feature masking, joint aggregation, and local frame-context aggregation on BOBSL validation data.Source: §4.3 · p. 7, Table 2 · p. 7, Table 3 · p. 7
4.4. Comparison to the state-of-the-art
- The fused pose-and-lip ISLR model outperforms Video-Swin-ISLR across metrics while using 1.05 GFLOPs versus 2.22 GFLOPs.The pose backbone alone reaches 75.2% Top-1 per-instance accuracy, while lip features complement pose features in late fusion.Source: §4.4 · p. 7, Table 4 · p. 7
- The multitask pretrained model achieves state-of-the-art BOBSL SLT, more than doubling LiTFiC (Full) on BLEU-4 using only video input, while SLT-only finetuning performs nearly identically.The no-pretraining variant also outperforms prior works, indicating strong representations from the visual backbone and Sliding Perceiver.Source: §4.4 · p. 7, Table 5 · p. 8
- On BOBSL SSA, multitask training improves alignment substantially, raising CB-Acc by 2.5% and F1@0.5 by 5.1 while maintaining comparable frame accuracy to SAT+.The SSA-only model improves F1@0.5 by 3.2 but has 1.5% lower frame accuracy than SAT+.Source: §4.4 · p. 8, Table 6 · p. 8
- The pretrained model surpasses prior zero-shot results on How2Sign and FLEURS-ASL, and finetuning on How2Sign surpasses previous methods despite using less pretraining data than SSLT.On How2Sign, the zero-shot model reaches B-RT 51.3; the finetuned model also performs comparably to a reported human baseline on selected metrics.Source: §4.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8
5. Conclusion
The paper concludes that combining pose and lip features produces generalisable sign-language embeddings supporting strong recognition, translation, and alignment across languages, datasets, and signers.
5. Conclusion
- The model achieves state-of-the-art ISLR, SLT, and SSA performance by combining pose and lip features into generalisable embeddings.Lip features capture fine-grained mouthing details, while the lightweight backbone supports end-to-end training with large-scale datasets and LLMs.Source: §5 · p. 8
- The 20-second signing window limits reliability when subtitles or audio-to-signing delays exceed that duration.The conclusion identifies longer or offset subtitle timing as a practical boundary for deployment.Source: §5 · p. 8
A. Implementation details
The implementation combines articulator-specific pose encoding, Conformer modules, visual–text projection, auxiliary supervision, Sliding Perceiver aggregation, DTW-based subtitle postprocessing, text cleaning, and class-balanced accuracy.
A.1. Pose backbone encoder
- The pose encoder uses articulator-specific AGCN processing, reshaping and fully connected joint fusion, with F = 24, Cin = 64, and Cout = 384.Face, body, and hand streams use 128, 33, and 21 joints respectively.Source: §A.1 · p. 13, Table A.1 · p. 13
A.2. Conformer
- Each Conformer maps its input to a shared hidden size of C′ = 512 through four blocks and attentive temporal pooling.Pose, lip, and auxiliary modules use input dimensions 1,536, 768, and 384 respectively.Source: §A.2 · p. 13, Table A.2 · p. 13
A.3. MLP projector for constrastive loss
- The MLP projector aligns visual and textual embeddings for contrastive ISLR pretraining, while the auxiliary articulator network adds pseudo-gloss supervision.The projector uses two fully connected layers with ReLU, and the auxiliary network applies shared-weight Conformers to articulator-specific features.Source: §A.3 · p. 13, Table A.3 · p. 13, Figure A.1 · p. 14
A.5. Sliding Perceiver
- The Sliding Perceiver projects features to D = 2,048, adds positional encoding, and summarizes stride-2 windows of size 8 through N = 2 cross-attention layers.Latents provide queries while local visual features provide keys and values.Source: Figure A.3 · p. 14, §A.5 · p. 14
A.6. DTW subtitle postprocessing for SSA
- DTW subtitle postprocessing uses plateau-decay probability profiles with β = 0.2 to resolve overlaps while preserving shorter subtitle sequences.The central 20% retains probability 1.0 and cosine decay defines the transition regions.Source: §A.6 · pp. 14–15, Figure A.4 · p. 15, Table A.4 · p. 15
A.7. Text cleaning procedures for YouTube-SL-25
- The YouTube-SL-25 cleaning pipeline removes markup, annotations, special symbols, speaker identifiers, excess spaces, and quotation marks.These operations target textual consistency before training.Source: §A.7 · p. 15
A.8. Class-balanced accuracy (CB-Acc)
- CB-Acc averages recall across background and subtitle classes so each class contributes equally despite differing durations.Classes comprise non-signing frames and one class for each subtitle.Source: §A.8 · p. 15
B.1. Ablation on late fusion ratio of visual backbone
The visual backbone performs best with a 0.7:0.3 pose-to-lip fusion ratio, while soft-decoding improves every reported SSA metric.
B.1. Ablation on late fusion ratio of visual backbone
- The 0.7:0.3 pose-to-lip logit fusion achieves the best overall ISLR performance.The ratio combines pose-backbone logits with lip-backbone logits.Source: §B.1 · p. 16
B.2. Effectiveness of soft-decoding
- Soft-decoding consistently improves all evaluation metrics when used for SSA training.The method combines cross-entropy with an L1 loss on soft-decoded and ground-truth timestamps.Source: §B.2 · p. 16
B.3. Analysis on efficiency of visual backbone
The visual backbone substantially reduces memory requirements while retaining practical joint-training feasibility, and the model wins against LiTFiC on most BOBSL test sentences.
B.3. Analysis on efficiency of visual backbone
- 29.6 GB versus over 96 GB of GPU memory is required for 500-frame processing by the proposed backbone and Video-Swin-ISLR, respectively.The proposed model can therefore be trained jointly with the LLM on a 48 GB GPU with batch size 1.Source: §B.3 · p. 16
- Training remains extremely time-consuming in the memory-feasible setting, with batch size constrained to 1 and about 280 hours needed for 400K iterations on an A6000.Source: §B.3 · p. 16
- 5.6 GB of memory enables batch size 4 and reduces training time to roughly 68 hours per 100K iterations after sliding-window processing is restricted to the Conformer.The AGCN encoder processes all 500 frames at once in this configuration.Source: §B.3 · p. 16
- Soft-decoding improves every SSA metric when combining cross-entropy with L1 timestamp loss.The ablation is reported on the BOBSL-SENT validation set.Source: Table A.6 · p. 16
B.4. Sentence-level BLEURT comparison with LiTFiC
- 13,647 of 20,338 BOBSL-SENT test sentences, or 67.1%, receive higher BLEURT scores from the model than from LiTFiC [41].This sentence-level comparison indicates higher BLEURT on most test samples.Source: §B.4 · p. 16, Table A.7 · p. 16
B.5. Ablation on all components
Ablations show that instruction-tuned decoding, stronger alignment supervision, lightweight pose-and-lip features, the Sliding Perceiver, end-to-end tuning, and beam search each contribute to translation quality.
B.5. Ablation on all components
- The complete ablation sequence raises B4 from 3.4 to 7.9 on BOBSL-SENT validation, approximately doubling performance.Table A.8 evaluates the incremental contribution of the model components.Source: Table A.8 · p. 17
- Replacing LLaMA-8B with FLAN-T5-XL improves all metrics, while DoRA and higher adaptation rank add further gains including +0.2 and +0.7 B4.The higher-rank configuration uses rank 32 and scaling 64.Source: §B.5 · p. 17
- Using SAT+ alignment supervision instead of SAT improves SLT performance, with alignment F1@0.50 increasing from 53.4 to 63.8.The result underscores the importance of higher-quality SSA supervision for translation.Source: §B.5 · p. 17
- Replacing Video-Swin-ISLR features with the proposed pose-and-lip backbone improves B4 by 0.5, while 20-second clips add another 0.4.The longer clips provide broader visual context.Source: §B.5 · p. 17
- The Sliding Perceiver adds +0.9 B4 over an MLP, end-to-end visual-backbone tuning adds 0.8, and beam search adds 0.5.Beam search uses beam size 5 and reduces repeated high-confidence outputs qualitatively.Source: §B.5 · p. 17
C. Qualitative results
Qualitative examples show stronger sign-language translation than LiTFiC on several BOBSL cases, while alignment examples show better span overlap than SAT+ despite similar frame-level accuracy.
C.1. Sign language translation
- Our model more accurately preserves meaning, numbers, and names than LiTFiC across several BOBSL translation examples.LiTFiC mispredicts “importing” as “exporting,” distorts long-sentence numbers and names, while our model remains closer to the ground truth.Source: §C.1 · pp. 17–18
- The model’s BOBSL translation failure on “jellyfish” versus “sharks” indicates that context beyond sign movements remains important.LiTFiC uses pseudo-glosses, the previous sentence, and background description in this example.Source: §C.1 · p. 18
C.2. Sign-subtitle alignment
- Around 40% span overlap is achieved by our model where SAT+ has none, although both models obtain the same frame-level accuracy.The qualitative comparison illustrates why frame-level accuracy can be inflated by background frames, whereas F1 better reflects alignment quality.Source: §C.2 · p. 18, Figure A.6 · p. 22
- Additional BOBSL cases show more accurate alignment predictions from our model than SAT+.The comparison includes further examples in the SSA qualitative-results figure.Source: §C.2 · p. 18, Figure A.6 · p. 22