Source-linked AI summary
SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval
Sobhan Asasi, Ozge Mercanoglu Sincan, Richard Bowden
TL;DR
Sign dictionary retrieval requires matching a query video to a dictionary entry across signer and recording variation, a setting not addressed by closed-set recognition. SignSeek learns pose-based representations with contrastive alignment and saliency-guided articulator masking. It reports state-of-the-art cross-corpus retrieval without fine-tuning, plus zero-shot BSL and transfer to recognition and subtitle alignment.
Problem
Sign dictionary retrieval has received little attention, and closed-set recognition does not require embeddings to generalize across unseen signers, signs, recording conditions, or signing styles.
Source: §1 · p. 1Method
SignSeek pretrains pose-based articulator streams with same-gloss contrastive alignment and saliency-guided masking for masked contrastive alignment and latent prediction.
Source: §1 · p. 1, §3.1 · p. 3, §3.3 · p. 4Results
Takeaways & Limitations
The same pose-based representations support cross-corpus dictionary retrieval and transfer to isolated recognition and subtitle alignment.
Source: §5 · pp. 8–9
Abstract
from arXiv · showhide
Sign language dictionaries are essential resources for sign language learners, yet automatically retrieving a sign from a dictionary, given only a query video, remains a challenging problem due to the natural variability between signers. Existing sign representation learning methods are built for closed-set recognition, producing embeddings that do not generalise to the open-set, signer-independent setting that retrieval demands. \textbf{SignSeek} closes this gap by contrastively learning sign representations with saliency-guided articulator masking. A contrastive objective aligns same-gloss signs across signers, while our Articulator Saliency-Guided Masking (ASGM) pinpoints the single most critical articulator per sign. This drives two complementary objectives, a masked contrastive alignment (MAC) loss that sees the sign through a single articulator and a masked prediction (MAP) loss that reconstructs it in latent space from the surrounding spatio-temporal context. Pretrained on 266K samples ($\sim$5,700 glosses) across multiple sign languages, \textbf{SignSeek} sets a new state-of-the-art performance in cross-corpus retrieval on ASL-Citizen, WLASL, and NMFs-CSL without any downstream fine-tuning. Strikingly, it achieves zero-shot generalisation to an entirely unseen British Sign Language (BSL), surpassing methods explicitly trained on BSL, and transfers seamlessly to isolated sign recognition and subtitle alignment, outperforming prior skeleton-based methods.
1. Introduction
Sign dictionary retrieval asks a model to match a query video to a dictionary entry, a transfer challenge that closed-set recognition does not address. SignSeek uses pose-based contrastive learning and saliency-guided articulator masking, achieving cross-corpus retrieval and transfer to other sign tasks without downstream fine-tuning.
1. Introduction
- Sign dictionary retrieval matches a query sign video to a dictionary entry, supporting education, documentation of linguistic variation, and accessibility tools.The task has received little attention compared with isolated sign recognition.Source: §1 · p. 1
- Unlike closed-set recognition, retrieval requires a metric space that clusters similar signs across unseen signers, recording conditions, and signing styles.Recognition discriminates among training classes, whereas retrieval tests transferability of representations.Source: §1 · p. 1
- Prior representations often target recognition or generic temporal data, overlooking distinct linguistic contributions of handshape, arm trajectory, and non-manual features.Pose keypoints abstract away background clutter and subject appearance, supporting signer-invariant representation learning.Source: §1 · p. 1
- SignSeek encodes face, body, and hand pose streams separately, then aligns same-gloss samples across signers with contrastive learning and articulator-aware masking.ASGM selects the most critical articulator; MAC reads the sign through it, while MAP predicts its latent representation from remaining context.Source: §1 · pp. 1–2
- SignSeek achieves state-of-the-art cross-corpus retrieval on ASL-Citizen, WLASL, and NMFs-CSL without fine-tuning, and transfers to unseen BSL, isolated recognition, and subtitle alignment.Pretraining uses 266K labelled samples spanning roughly 5,700 glosses across four languages.Source: §1 · p. 2
2. Related Work
Prior work largely approaches sign dictionary retrieval through recognition embeddings or as a secondary evaluation, while related retrieval, alignment, and representation-learning tasks use different objectives. SignSeek instead centers open-set dictionary retrieval and combines pose representations with articulator-level masking.
Sign Language Representation
- Pose-based recognition methods use skeleton keypoints and graph networks, but prior methods target closed-set recognition rather than retrieval embeddings.SignSeek operates on pose, needs only gloss labels, and models articulatory structure for open-set retrieval.Source: §2 · p. 2
- The multilingual pretraining collection combines isolated-sign datasets with continuous MeinDGS clips segmented by gloss, spans four languages, and has no gloss overlap.The caption notes that glosses shared with WLASL and their samples are removed from MSASL.Source: Table 1 · p. 2
Sign Dictionary Retrieval (SDR)
- Sign dictionary retrieval matches an isolated sign query to a gloss-level entry, whereas sign language retrieval matches continuous sentence-level signing to free-form text.Prior work often treats dictionary retrieval as a recognition by-product or secondary metric; SignSeek benchmarks it as a primary task under a zero-shot protocol.Source: §2 · p. 2
Sign-Subtitle Alignment (SSA)
- Subtitle alignment methods may train specifically for alignment, while SignSeek reuses isolated-sign retrieval embeddings and dynamic programming without alignment-specific training.Earlier pipelines commonly segment continuous signing before aligning segments using audio timings or learned segmentation.Source: §2 · p. 2
Contrastive and Masked Representation Learning
- Figure 1 shows query and reference clips passing through a shared per-articulator encoder, temporal fusion, and pooling before three objectives operate on their embeddings.GAC uses intact clips, MAC masks the query to its dominant articulator, and MAP masks that articulator in the reference and predicts its clean representation.Source: Figure 1 · p. 3
- SignSeek replaces global augmentation with articulator-level masking, combining contrastive alignment over masked articulators with latent prediction of the dominant articulator.Prior masked-modelling approaches reconstruct masked inputs or predict representations in a learned latent space.Source: §2 · pp. 2–3
3. Methodology
SignSeek represents signs as separate articulator streams, scores each articulator’s saliency, and uses the selected articulator to guide complementary contrastive and prediction objectives. At inference, it uses the intact-stream clip embedding for retrieval.
3.1. Articulatory Representation
- SignSeek represents each clip as separate body, face, left-hand, and right-hand streams of 2D keypoints and confidence scores.Each stream has T frames; its keypoint tensor includes joint coordinates and estimation confidence.Source: §3.1 · p. 3
3.2. Articulatory Backbone
- Dedicated graph encoders process the body and face, while a shared hand encoder mirrors the right hand before temporal fusion and attentive pooling produce a clip embedding.The articulator streams are concatenated and passed through a Conformer that captures long-range temporal structure.Source: §3.2 · pp. 3–4
3.3. Articulator Saliency-Guided Masking (ASGM)
- ASGM scores standardized, temporally pooled articulator representations and samples one dominant articulator per clip to guide the training objectives.Standardization makes scores comparable across streams without auxiliary supervision or learned parameters; Gumbel-max sampling preserves saliency ordering in expectation.Source: §3.3 · p. 4
3.4. Training Objectives
- GAC pulls intact embeddings of same-gloss clips together and pushes other glosses apart, while MAC applies this contrastive alignment to a query showing only its dominant articulator.Same-gloss positives are defined across clips and signers; MAC requires the dominant-only query to align with clean same-gloss references.Source: §3.4 · pp. 4–5
- MAP masks the dominant stream and predicts the intact clip’s latent sequence from the remaining context, encouraging recovery of its contribution to sign meaning.The prediction target is held fixed by stop-gradient, and the loss aligns prediction and target frame-wise over valid frames.Source: §3.4 · p. 5
- At inference, SignSeek embeds intact streams and ranks dictionary entries by cosine similarity, without using masking, ASGM, or the predictor.The clip embedding comes from the graph encoders, temporal fusion Conformer, and attentive pooling head.Source: §3.4 · p. 5
4.1. Datasets and Evaluation Protocol
The frozen encoder is evaluated on retrieval datasets disjoint from pretraining, in an unseen-language BSL dictionary lookup, and on downstream recognition and subtitle alignment tasks. The protocol specifies rank-based retrieval metrics and task-specific downstream metrics.
4.1. Datasets and Evaluation Protocol
- The encoder is pretrained on five corpora across four languages, with 266K labelled clips and roughly 5,700 glosses, then frozen for all evaluations.Training combines isolated-sign datasets with gloss-segmented clips from continuous MeinDGS.Source: §4.1 · p. 5
- Retrieval is evaluated zero-shot on ASL-Citizen, WLASL, and NMFs-CSL, which are disjoint from pretraining, and on BSL, an entirely unseen language.The BSL lookup uses 726 query–gloss pairs from a native signer against the BSL SignBank dictionary; the encoder is also applied without fine-tuning to subtitle alignment on How2Sign and BOBSL.Source: §4.1 · pp. 5–6
- Retrieval ranks glosses by the maximum cosine similarity to their gallery instances and reports Recall@k, mAP, and DCG; downstream tasks use Top-1/Top-5 accuracy and F1@0.5.The retrieval gallery is the training split and queries come from the test split.Source: §4.1 · p. 6
4.2. Implementation Details
Training uses a fixed schedule, balanced gloss sampling, and equally weighted objectives.
4.2. Implementation Details
- Training runs for 30,000 steps with AdamW, learning rate 3 × 10^-4, weight decay 0.01, and 3,000 warmup steps.Full hyperparameters are provided in the supplementary material.Source: §4.2 · p. 6
- Each batch samples 40 glosses with two clips per gloss using balanced sampling.Source: §4.2 · p. 6
- The contrastive and ASGM selection temperatures are 0.07 and 0.7, respectively, and the training objectives are weighted equally.Source: §4.2 · p. 6
4.3. Results
SignSeek improves over the strongest retrieval baseline across three benchmarks and also generalizes to unseen BSL. Its WLASL embeddings form more separated gloss clusters than the compared representations.
4.3. Results
- SignSeek improves over SignRep on every metric and dataset, raising DCG from 71.21 to 77.21 on ASL-Citizen, 57.93 to 61.24 on WLASL2000, and 83.05 to 85.17 on NMFs-CSL.The benchmark comparison uses frozen features without fine-tuning, except MASA, which saw WLASL2000 and NMFs-CSL during training.Source: §4.3 · p. 6
- On unseen BSL, SignSeek reaches 12.76 R@1 and 20.84 mAP, surpassing BSL-trained Video-Swin by 3.61 R@1 and 7.47 mAP.Evaluation uses 726 native-signer clips against the BSL SignBank dictionary; SignSeek uses body pose and a frozen 15.7M-parameter extractor.Source: §4.3 · p. 7
- SignSeek forms tighter, better-separated clusters for the ten most frequent WLASL glosses than Video-Swin and SignRep, alongside higher 1-NN accuracy.Figure 2 titles report the percentage of samples whose nearest neighbour shares the gloss.Source: Figure 2 · p. 6, §4.3 · p. 7
4.4. Sign-Subtitle Alignment (SSA)
SignSeek uses frozen embeddings and dictionary-derived gloss prototypes to align subtitles with signing through dynamic programming. It achieves the best How2Sign F1@0.5 and remains competitive with SignCLIP on BOBSL.
4.4. Sign-Subtitle Alignment (SSA)
- For each subtitle pseudogloss, SignSeek retrieves language-matched dictionary exemplars, encodes prototypes and video windows, then decodes a monotonic alignment path with dynamic programming.The gloss-window similarity is the maximum cosine similarity between a window embedding and that gloss’s prototype set.Source: §4.4 · p. 7
- SignSeek achieves the best F1@0.5 on How2Sign and remains competitive with SignCLIP on BOBSL, improving over SignCLIP by 1.0 on validation and 2.7 on test.On BOBSL, the reported values are 66.94 to 66.70 and 50.72 to 50.68.Source: §4.4 · p. 7
- The SignSeek encoder remains frozen and needs no alignment-specific training; its pose pretraining uses gloss labels, unlike baseline representations requiring boundary annotations or paired sign-text.All compared methods reuse general representations and recover alignments with dynamic programming.Source: §4.4 · p. 7
4.5. Isolated Sign Language Recognition (ISLR)
Fine-tuning SignSeek for isolated sign recognition makes it the strongest skeleton-based method across the three evaluated datasets.
4.5. Isolated Sign Language Recognition (ISLR)
- SignSeek is the strongest skeleton-based method on all three datasets after adding a classification layer and fine-tuning the whole model.It reaches 56.65% Top-1 instance accuracy on WLASL, 7.6 points above the best prior skeleton model, and improves Top-1 by 2.1 points on NMFs-CSL and 14.7 points on ASL-Citizen over the stated comparators.Source: §4.5 · p. 7
4.6. Ablation Studies
The ablations show that combining global contrastive alignment with both articulator-masking objectives performs best, while saliency-focused selection and balanced, diverse training batches support retrieval performance.
4.6. Ablation Studies
- 34.75 WLASL R@1 and 57.66 ASL-Citizen R@1 are reached by combining GAC, MAC, and MAP, exceeding either masked objective alone.MAC alone reaches 31.27 and 54.15; MAP alone reaches 32.33 and 55.93 on WLASL and ASL-Citizen, respectively.Source: §4.6 · p. 7, Table 7 · p. 8
WLASL ASL-Citizen
- 34.75 WLASL R@1 and 57.66 ASL-Citizen R@1 at τ=0.7 fall toward random-selection performance as temperature rises.At τ=2.0, R@1 is 30.15 on WLASL and 53.16 on ASL-Citizen, approaching the random-selection baseline of 29.66 and 52.22.Source: WLASL ASL-Citizen · p. 8, Table 8 · p. 8
- Fixed articulator choices lower retrieval: face-only selection reaches 24.78 R@1 on WLASL, while face and body are the strongest fixed choices on NMFs-CSL.On NMFs-CSL, fixed face and body choices reach 61.63 and 59.23 R@1, versus 56.10 for right-hand-only selection.Source: WLASL ASL-Citizen · p. 8, Table 9
- 34.8 WLASL R@1 and 57.7 ASL-Citizen R@1 at k=40, n=2 exceed results at k=10, n=8, with larger k improving all three benchmarks.Removing class balancing hurts retrieval on every dataset; Figure 4a presents the batch-composition comparison.Source: WLASL ASL-Citizen · p. 8, Figure 4 · p. 8
- Retrieval improves as the per-gloss cap grows from 3 to 10 clips, then plateaus at full-model performance without further gains from larger caps.Figure 4b shows saturation by about 10 clips per sign and a peak near 30.Source: WLASL ASL-Citizen · p. 8, Figure 4 · p. 8
5. Conclusion
SignSeek is a pose-based pretraining framework that encodes articulatory structure through saliency-guided masking. It achieves state-of-the-art cross-corpus retrieval without fine-tuning and transfers to unseen BSL, isolated sign recognition, and subtitle alignment.
5. Conclusion
- SignSeek is a pose-based pretraining framework that treats sign dictionary retrieval as a primary objective and uses saliency-guided articulator masking.The objectives recognize a sign through its dominant articulator and recover it from surrounding context.Source: §5 · p. 8
- SignSeek sets a new state of the art in cross-corpus retrieval across ASL-Citizen, WLASL, and NMFs-CSL without fine-tuning.Source: §5 · p. 8
- SignSeek generalises to entirely unseen British Sign Language and transfers to isolated sign recognition and subtitle alignment.Source: §5 · pp. 8–9